Contents
Figure 1: OR-Tools vs Reinforcement Learning: When a Solver Actually Wins — the solver side is cheap to measure, and that measurement is the whole argument
Your OR-Tools model solves a 200-stop routing problem in under a minute. Then a colleague says "let's try reinforcement learning," and six weeks later you have a neural network that loses to the solver. Sound familiar?
I have not run a head-to-head benchmark against a trained RL policy, so treat the RL side of this post as analysis, not measurement. The solver side I did measure, on a 200-stop instance, and those numbers are below. My honest read: the answer is "rarely, but in specific situations, decisively."
What Are We Comparing?
OR-Tools is Google's open-source optimization suite: CP-SAT for constraint programming, a routing library for TSP and VRP, and linear and mixed-integer tools [1][2]. It searches for a solution to the specific instance in front of it, every time you run it.
Reinforcement learning trains an agent by trial and error in a simulator, and that agent then applies its learned policy to new inputs. One approach optimises per instance; the other learns a general strategy. That single difference drives everything below.
The Solver Baseline I Measured
Before arguing about RL, it helps to know how strong the thing you would replace actually is. I generated a 200-stop capacitated vehicle routing problem: random points in a 100 × 100 km square, demands 1 to 10, 20 vehicles with capacity 100, seed 42, distances rounded to metres. Setup: OR-Tools 9.15.6755, Python 3.12, PATH_CHEAPEST_ARC first solution, guided local search, one core, no tuning beyond those two settings.
| Approach | Cost (m) | Wall time | Gap vs 30 s run |
|---|---|---|---|
| Greedy nearest neighbour | 2,708,310 | 2.7 ms | +23.9% |
| OR-Tools GLS, 0.2 s | 2,438,361 | 0.2 s | +11.5% |
| OR-Tools GLS, 0.5 s | 2,264,940 | 0.5 s | +3.6% |
| OR-Tools GLS, 1 s | 2,221,356 | 1.0 s | +1.6% |
| OR-Tools GLS, 5 s | 2,202,250 | 5.0 s | +0.7% |
| OR-Tools GLS, 30 s | 2,186,042 | 30.0 s | best found |
Two things stand out. A greedy construction — which is roughly what an untrained policy starts from — is 24% worse than a thirty-second search. And a one-second search lands within 1.6% of it, so the bar for "RL beats the solver" is not "produce a route," it is "produce a route within a couple of percent at a fraction of the latency."
The opening claim is, if anything, conservative: this instance is close to solved in one second on a single core, and thirty seconds buys 0.7% more. That 1.6% gap is the number a learned policy has to beat, and it has to beat it after paying for a simulator, training runs and hyperparameter search.
from ortools.constraint_solver import pywrapcp, routing_enums_pb2
manager = pywrapcp.RoutingIndexManager(len(dist), vehicles, 0)
routing = pywrapcp.RoutingModel(manager)
routing.SetArcCostEvaluatorOfAllVehicles(routing.RegisterTransitCallback(arc_cost))
routing.AddDimensionWithVehicleCapacity(
routing.RegisterUnaryTransitCallback(demand_of), 0, [100] * vehicles, True, "Capacity"
)
params = pywrapcp.DefaultRoutingSearchParameters()
params.first_solution_strategy = routing_enums_pb2.FirstSolutionStrategy.PATH_CHEAPEST_ARC
params.local_search_metaheuristic = (
routing_enums_pb2.LocalSearchMetaheuristic.GUIDED_LOCAL_SEARCH
)
params.time_limit.FromSeconds(1)
solution = routing.SolveWithParameters(params)
print(int(solution.ObjectiveValue()))
Where OR-Tools Wins
Start with the uncomfortable truth for RL fans: a solver wins most head-to-head fights on a single problem.
Static, One-Off Problems
If you solve a problem once, or the data barely changes, a solver returns a near-optimal answer with no training at all. You write a model, press run, and get a result. RL asks you to build a simulator, train for hours or days, and tune hyperparameters first. Paying that cost for one answer is hard to justify.
Hard Constraints
Time windows, capacities, precedence rules and legal limits need guaranteed feasibility. The routing library models these as dimensions, and CP-SAT handles arbitrary constraints natively [1][2]. An RL agent learns constraints softly, so it can violate them unless you add action masking or a repair step — and a repair step is a solver in disguise.
Optimality Proofs and Bounds
Mixed-integer and constraint solvers can report optimality gaps and bounds, so you know how far an answer sits from the best possible one. RL returns a number with no certificate, which makes auditing and sign-off harder.
Small and Medium Instances
For dozens to a few hundred decision points, classical heuristics and metaheuristics are extremely strong, as the table above shows. Decades of engineering went into them, and research papers often report RL gains against weaker baselines — check which baseline a paper actually used before believing the headline.
Where RL Can Win
RL earns its keep in a narrower set of cases. These are the ones I find convincing.
Uncertainty That Never Stops
Delivery requests arrive at random, travel times fluctuate, machines break down. A static solver needs a re-solve every time the world changes; a trained policy reacts immediately and, because it trained on simulated randomness, can anticipate it rather than only respond.
Massive Repetition
Suppose you solve thousands of similar instances daily. Training once and reusing the policy amortises the upfront cost, inference takes milliseconds, and a solver may need seconds per instance. The arithmetic only works at volume, so estimate your volume before the pitch meeting.
Tight Latency Budgets
Real-time dispatch, ad allocation and cluster scheduling leave almost no time to think. A forward pass fits where a search does not, and a fast decent answer beats a perfect late one — which is exactly what the 0.2 s row above shows a solver doing badly.
Problems Without Clean Models
Some systems resist formulation: the dynamics are complex, but you can simulate them. RL learns directly from the simulator, while a solver needs equations you cannot easily write. This is the strongest case for RL, and the weakest case for a benchmark comparison, because the solver has nothing to solve.
Head-to-Head Comparison
| Dimension | OR-Tools | Reinforcement learning |
|---|---|---|
| Setup effort | Model the problem | Simulator, training, tuning |
| Quality on one instance | Usually wins | Needs training to compete |
| Speed per decision | Seconds to minutes | Milliseconds after training |
| Uncertainty | Re-solve each change | Reacts from the policy |
| Constraint guarantees | Native and exact | Soft, needs masking or repair |
| Optimality proofs | Gap and bounds available | None |
| Generalisation | No retraining needed | Can fail off-distribution |
| Explainability | Model is inspectable | Policy is a black box |
Notice which side wins more rows. That is not an accident, and it is why the honest default for routing with hard constraints is still a solver.
The Hybrid Option Most Teams Overlook
You do not have to choose. The practical designs combine both:
- Warm starts: a learned policy proposes a solution, and OR-Tools polishes it.
- Learned heuristics: RL picks which local search move to try next.
- Learned branching: neural networks guide branch-and-bound decisions [5].
- Predict-then-optimise: a model forecasts demand, and a solver plans around it.
Hybrids keep solver guarantees and add learned speed or adaptability, and I would start there for any production system with hard constraints.
But warm starts are not a free win, and I measured that too. Same instance, same settings, greedy routes fed in as the initial solution:
| Budget | Plain solve | Greedy warm start | Delta |
|---|---|---|---|
| 0.2 s | 2,438,361 | 2,442,221 | −0.16% |
| 0.5 s | 2,264,940 | 2,276,791 | −0.52% |
| 1 s | 2,244,209 | 2,221,232 | +1.02% |
| 5 s | 2,202,250 | 2,202,250 | 0.00% |
The warm start helped at one second, slightly hurt at 0.2 and 0.5 seconds, and changed nothing at five. Time-limited guided local search is also not deterministic: the same one-second budget produced 2,244,209 in one run and 2,221,356 in another, so treat any single-run delta under about 1% as noise and repeat your runs.
That is the real lesson for hybrids. A warm start buys you a better starting point, and guided local search mostly cares about how long it searches, not where it starts. Measure the combination on your own instances before you build a pipeline around it.
A Fair Way to Run the Test
Want to settle the question on your own problem? Follow a protocol that does not flatter either side:
- Define the metric: cost, makespan or profit, plus runtime.
- Set the same wall-clock budget for both, and remember that the RL policy's "milliseconds" exclude its training.
- Tune the solver seriously: try several metaheuristics and parameters before you call it a baseline.
- Test across sizes and distributions, including instances larger than the training data.
- Include disruptions: arrivals, delays, breakdowns, because that is where RL should shine.
- Count total cost: training time, engineering effort, infrastructure and maintenance.
A weak baseline makes any method look brilliant. Tune the solver like you want it to win, then see what survives.
Common Pitfalls
- Comparing against an untuned solver. Defaults undersell OR-Tools badly — the table above uses two settings, and parameter tuning moves it further.
- Ignoring training cost. A policy that took two weeks to train did not "run in milliseconds" in any honest accounting.
- Testing only on training-like data. Generalisation failures hide until deployment.
- Skipping feasibility checks. An RL route that violates a time window is not a route.
- Using a toy simulator. The agent masters your simulator, not the world.
- Reporting a single run. Time-limited search varies between runs, so a 1% "win" from one seed is not a result.
Every one of those mistakes makes RL look better than it is, which is why so many six-week projects end with a model that loses to the solver it replaced.
A Quick Decision Guide
Ask these in order:
- Is the problem static or rarely repeated? Use OR-Tools.
- Are hard constraints critical? Use OR-Tools, or a hybrid with masking and repair.
- Does it repeat thousands of times under real uncertainty? Consider RL.
- Do you need decisions in milliseconds? Consider RL, or a hybrid with a solver polishing the policy's output.
- Can you build a simulator you trust? If not, skip RL entirely.
Best Book to Read
| Book | Description | Get it |
|---|---|---|
| Algorithms for Decision Making | covers Markov decision processes, reinforcement learning and optimisation in one framework, so you can see why a sequential decision problem and a one-shot combinatorial problem deserve different tools [3]. | View on Amazon |
As an Amazon Associate I earn from qualifying purchases. Links on this page are affiliate links, and I may earn a commission at no extra cost to you.
For the RL half specifically, Sutton and Barto's textbook is free online and remains the reference [4].
Wrapping Up
OR-Tools beats RL in most fair fights on a single, well-defined problem: within 1.6% of a thirty-second solution in one second, on the 200-stop instance measured here. RL wins when uncertainty, repetition and speed dominate, and when you can simulate the world well enough to train in it. The strongest systems usually run both, with the solver keeping the guarantees.
So run the experiment this week. Take one real problem, tune OR-Tools properly, and build a small RL baseline against it. If the solver wins, you saved yourself a risky rewrite. If RL wins, you found a rare gem — and you will know it from your own numbers, not from a paper's.
Frequently Asked Questions
Is reinforcement learning faster than OR-Tools for routing?
Only after training, and only in the regime where per-instance solve time dominates. On the 200-stop instance measured for this post, OR-Tools reached within 1.6% of a thirty-second solution in one second. An RL policy has to beat that gap at lower latency before the training cost and the simulator become worth it.
Can OR-Tools handle time windows and capacity constraints?
Yes, natively. The routing library models time windows, capacities, pickup and delivery pairs, and precedence as dimensions, and CP-SAT handles arbitrary constraint programming. An RL agent learns constraints softly, so it needs action masking or a repair step to guarantee feasibility.
When does reinforcement learning actually beat a solver?
When the problem repeats thousands of times with genuine uncertainty, when decisions must arrive in milliseconds, or when the system is easier to simulate than to formulate. Sequential stochastic control with no clean mathematical model is the strongest case.
What is a warm start in vehicle routing?
A feasible solution handed to the solver as a starting point, usually from a greedy construction or a learned policy. It is not a free win: in the measurements for this post a warm start helped at a one-second budget and slightly hurt at 0.2 and 0.5 seconds.
How should I benchmark a solver against an RL policy fairly?
Give both the same wall-clock budget, tune the solver's metaheuristics instead of using defaults, test on instances larger than the training distribution, include disruptions, and count training time, engineering effort and maintenance. A weak baseline makes any method look brilliant.
Sources
[1]Google OR-Tools, Routing and vehicle routing — the routing library documentation, including the search parameters, first-solution strategies and local search metaheuristics used in the benchmark: developers.google.com/optimization/routing[2]Google OR-Tools, CP-SAT solver — the constraint programming documentation, covering the exact constraints and optimality bounds that a learned policy cannot certify: developers.google.com/optimization/cp[3]Kochenderfer, Wheeler and Wray, Algorithms for Decision Making — the authors' site for the MIT Press textbook, covering MDPs, reinforcement learning and optimisation in one framework: algorithmsbook.com[4]Sutton and Barto, Reinforcement Learning: An Introduction, 2nd edition — the free online edition, still the reference for policy learning and the exploration/exploitation trade-off: incompleteideas.net/book/the-book-2nd.html[5]Bengio, Lodi and Prouvost, Machine Learning for Combinatorial Optimization: a Methodological Tour d'Horizon — the survey that separates learning-to-configure, learned heuristics and end-to-end policies, and documents the baseline problems in the literature: arxiv.org/abs/1811.06128