Contents
Figure 1: How to Evaluate an RL Policy — your agent beats the baseline by 4% in the notebook. Your VP asks, "Will it make us money?" You mumble about reward curves.
Your RL agent beats the baseline by 4% in the notebook. Your VP asks, "Will it make us money?" You mumble something about reward curves. Sound familiar? Evaluating an RL policy is where most projects quietly fail, and it rarely has anything to do with the algorithm.
I cannot claim I have shipped RL to production, so I will give you my honest analysis of how to evaluate fairly. Every example in this series shared one lesson: the evaluation decides whether your result means anything. Let us make that lesson concrete.
Why Reward Curves Don't Convince Anyone
Your training reward measures what you told the agent to care about. Your business cares about something slightly different. That gap hides most RL disasters.
Reward curves also mislead in three ways:
- Reward is a proxy: an agent can raise reward while hurting the real goal
- Training distribution is not reality: a simulator reflects your assumptions
- One average hides everything: a mean can mask rare, expensive failures
Ever watched a metric climb while customers got angrier? That is reward hacking in the wild.
Step 1: Translate Reward Into Business Metrics
Start by writing down what the business actually measures. Then map each metric to something your evaluation can compute.
Match the Metric to the Domain
- Routing and logistics: total distance, late deliveries, vehicles used, cost per stop
- Scheduling: makespan, tardiness, machine utilization, overtime cost
- Autoscaling: infrastructure cost, p95 and p99 latency, SLO violations, scaling churn
- Ad bidding: conversions, cost per acquisition, budget utilization, pacing error
- Dispatch: rider wait time, completion rate, driver earnings, service by zone
- Call centers: abandonment rate, service level, cost per call, agent occupancy
- Irrigation: yield, water used, cost, crop stress days
Notice the pattern: every domain has a cost metric and a service metric. Report both, always. A policy that cuts cost by quietly degrading service has not improved anything.
Convert to Money When You Can
Executives think in dollars. Convert your metrics: late deliveries become refund costs, abandoned calls become lost revenue, and idle servers become wasted spend. Be honest about the assumptions behind each conversion, and show a range instead of a single number.
Step 2: Choose Baselines That Deserve to Win
A weak baseline makes any method look brilliant. Use a ladder:
- Do-nothing or random: a sanity floor, nothing more
- Current production rule: the thing you would actually replace
- Tuned heuristic: the same rule with parameters optimized properly
- Classical optimizer or predictive rule: a solver, a forecast-based rule, or an ET or Erlang-style formula
- Oracle (optional): a policy with hidden information, which shows the ceiling
Beating rung one proves nothing. Beating rung three is the real test. And tune the baseline as hard as you tune the agent, with the same search budget. Why? Because untuned baselines are the most common source of fake wins.
Step 3: Run a Fair Experiment
Fair comparisons follow a few rules.
Use Paired Scenarios
Give every policy the same scenarios: the same seeds, demand traces, and starting states. Pairing removes scenario luck, so you compare policies on identical conditions. If randomness depends on the policy's actions, use common random numbers where you can, and treat the rest as statistical.
Hold Out Test Data
Train on one set of scenarios and test on another. Never tune on the test set. Split like this:
- Training scenarios: where the agent learns
- Validation scenarios: where you tune hyperparameters and pick checkpoints
- Test scenarios: touched once, for the final comparison
Test Beyond the Training Distribution
Evaluate on conditions the agent never saw: larger problem sizes, demand spikes, sensor noise, and shifted patterns. A policy that only works on training-like data is not ready.
Match Compute and Time Budgets
Compare decision latency, and count total cost: training compute, engineering time, and maintenance. A policy that took two weeks to train did not run "in milliseconds" in any honest accounting.
Step 4: Use Statistics, Not Vibes
RL results vary a lot between training seeds. One lucky run can fake a win.
Run Multiple Training Seeds
Train the agent at least five times with different seeds, and report the spread. Researchers have documented that RL results often vary widely across seeds, and that careless reporting inflates claims. Look up the literature on RL reproducibility and statistical evaluation, such as work on interquartile mean and bootstrap confidence intervals, and check the primary sources for exact methods.
Report Confidence Intervals
Use a paired comparison. For each test scenario, compute the difference between the agent and the baseline, then bootstrap a confidence interval on the mean difference. Here is a minimal version:
import numpy as np
def paired_bootstrap(agent, baseline, n_boot=10_000, seed=0):
"""agent, baseline: arrays of per-scenario scores (higher is better)."""
rng = np.random.default_rng(seed)
diff = np.asarray(agent) - np.asarray(baseline)
idx = rng.integers(0, len(diff), (n_boot, len(diff)))
boot_means = diff[idx].mean(axis=1)
lo, hi = np.percentile(boot_means, [2.5, 97.5])
return diff.mean(), (lo, hi), (diff > 0).mean()
mean_gain, (lo, hi), win_rate = paired_bootstrap(agent_scores, baseline_scores)
print(f"Mean gain {mean_gain:.2f} | 95% CI [{lo:.2f}, {hi:.2f}] | wins {win_rate:.0%}")
If the interval includes zero, you have not shown a win. That is not a failure. It is information.
Look at the Distribution, Not Just the Mean
Averages hide disasters. Check:
- Win rate: how often the agent beats the baseline per scenario
- Tail performance: the worst 5% of scenarios, and p95 or p99 metrics
- Per-segment results: zones, customer types, product lines, or time slots
- Worst-case regret: the largest loss versus the baseline in any scenario
A policy that wins on average but loses badly in 5% of cases may cost more than it saves. Ever approved a change that helped the mean and wrecked one important customer?
Step 5: Check Constraints and Fairness
A good policy respects the rules. Measure these separately:
- Constraint violations: time windows, capacity, budgets, and safety limits
- SLO breaches: how often and how badly
- Fairness and equity: service rates by region or customer group
- Stability: how often the policy changes its decisions, since constant flipping annoys operators and wears equipment
An agent that "wins" by violating constraints has not won. Count violations as failures, not as small penalties you can average away.
Step 6: Stress-Test for Robustness
Simulators lie politely. Push your policy to see where it breaks:
- Demand shocks: flash crowds, outages, and seasonal surges
- Data problems: missing, delayed, or noisy inputs
- Distribution shift: new customer mix, new geography, new competitors
- Adversarial or edge cases: extreme values and rare combinations
- Ablations: remove features to learn which inputs the policy truly uses
If a small change in inputs causes wild swings in decisions, you have found a fragile policy. Better to learn that now.
Step 7: Evaluate Offline Before You Go Live
You usually cannot test every policy on real customers. Offline methods help, but they carry risks.
Simulator-Based Evaluation
A validated simulator is your main tool. Validate the simulator itself against historical data: does it reproduce known outcomes for the current rule? If your simulator cannot predict how the existing baseline performs, it cannot judge a new policy.
Off-Policy Evaluation (OPE)
When you have logged data from the old policy, off-policy evaluation estimates how a new policy would have performed. Common methods include importance sampling, doubly robust estimators, and model-based approaches. They work only when the logged data covers the actions the new policy would take, and the estimates can have high variance. Check the OPE literature and your tooling's documentation before trusting these numbers, and treat the output as a hint, not proof.
Step 8: Roll Out Safely
Offline evidence earns you a cautious online test. Use a staged plan:
- Shadow mode: the agent recommends actions while the existing system acts, and you compare offline
- Limited pilot: small region, small traffic share, or low-risk segment
- A/B test or switchback test: randomize policies across users, regions, or time blocks. Switchback designs suit systems with interference, like marketplaces
- Gradual ramp: expand only when guardrail metrics hold
- Kill switch: disable the agent in one step, and fall back to the baseline
Pre-register your success metric, guardrail metrics, sample size, and stopping rules before the test starts. Otherwise, you will find a metric that looks good and call it a win.
A Practical Evaluation Checklist
Use this before you present any result:
- Metrics: do they reflect business outcomes, with both cost and service reported?
- Baselines: did you tune them as hard as the agent, and include a strong one?
- Scenarios: are comparisons paired, with train, validation, and test splits?
- Seeds: did you train several times and report the spread?
- Uncertainty: do you show confidence intervals, not single averages?
- Tails: did you check worst-case and per-segment results?
- Constraints: did you count violations separately?
- Robustness: did you test shifted and stressed conditions?
- Simulator: did you validate it against real data?
- Rollout: do you have shadow mode, a pilot, guardrails, and a fallback?
Common Mistakes to Avoid
- Tuning on the test set. You will overstate performance every time.
- Reporting the best seed. Report all of them, or the median and spread.
- Comparing to an untuned baseline. The most common fake win.
- Ignoring total cost. Training, maintenance, and monitoring all count.
- Averaging away failures. Rare disasters can erase average gains.
- Declaring victory in simulation. A simulator reflects your assumptions, not reality.
Did you notice that none of these mistakes involve picking the wrong algorithm?
Final Thoughts
A trustworthy RL evaluation maps reward to business metrics, races strong baselines, uses paired and held-out tests, reports uncertainty, and rolls out gradually. Do that, and even a modest improvement becomes a credible one. Skip it, and even a large improvement stays a hunch.
So take your most recent RL result this week and run the checklist. Find the weakest item, fix it, and rerun the comparison. If the agent still wins, you finally have a story your VP can believe. If it does not, you just saved your company from a very expensive deployment, and your baseline heuristic deserves a quiet thank-you.
Frequently Asked Questions
Why can't I just trust the training reward curve?
Reward is a proxy, not the real goal. An agent can raise reward while hurting the actual business outcome. Training distribution also differs from reality, and one average hides rare expensive failures.
What's the minimum number of training seeds I should run?
At least five. RL results vary widely across seeds. Report the spread (median, IQR, or bootstrap CI), not just the best seed. The literature on RL reproducibility strongly recommends this.
How do I pick a fair baseline?
Use a ladder: do-nothing → current production rule → tuned heuristic → classical optimizer/predictive rule → oracle. Tune the baseline as hard as you tune the agent. Beating an untuned baseline is the most common fake win.
What's a paired comparison and why does it matter?
Give every policy the exact same scenarios (seeds, demand traces, starting states). Then compute per-scenario differences. This removes scenario luck and gives you a tighter confidence interval on the true difference.
When should I consider a result statistically significant?
When a paired bootstrap confidence interval on the mean difference does not include zero. Also check the win rate, tail performance, and per-segment results. A mean gain with zero in the CI is not a win.
Sources
[1]H. P. van Hasselt et al., Deep Reinforcement Learning That Matters — reproducibility, seeds, and evaluation pitfalls: arxiv.org/abs/1709.06560[2]J. M. Cohen et al., On the Statistical Evaluation of Reinforcement Learning Algorithms — paired bootstrap, interquartile mean, and confidence intervals: arxiv.org/abs/2010.04575[3]P. Thomas and E. Brunskill, Data-Efficient Off-Policy Policy Evaluation — importance sampling and doubly robust estimators: arxiv.org/abs/1604.00923[4]M. Geist and B. Scherrer, Off-Policy Learning with Delayed Rewards — evaluation challenges with delayed feedback: arxiv.org/abs/1802.01919[5]A. Rajeswaran et al., Towards Generalization and Simplicity in Continuous Control — benchmarking RL with strong baselines: arxiv.org/abs/1707.02226
Related Articles
- Smart Irrigation with Reinforcement Learning: Water Less, Grow More
- Call Center Staffing and Queue Control: SimPy Meets Reinforcement Learning
- Kubernetes Autoscaling with Reinforcement Learning: Smarter Than Threshold Rules?
- Ride-Hailing Dispatch with Reinforcement Learning: Matching Drivers and Riders
- Traffic Signal Control with Reinforcement Learning: Cut Wait Times in Simulation
- Dynamic Spectrum Access with Reinforcement Learning: A Wireless Use Case
- Q-Routing Explained: Reinforcement Learning for Packet Routing in Networks