Contents
Figure 1: Traffic Signal Control with Reinforcement Learning — a Manhattan intersection where the fixed-time plan has no idea the cross street is empty
You sit at a red light. No cars cross the intersection. Nobody moves. You wonder whether a smarter signal could fix this, and you are right. Reinforcement learning can learn signal timing from experience, and you can test it in simulation without annoying a single real driver.
I cannot claim I have deployed RL on a real corridor, so treat this as a learning project. I did run the simulator below on a laptop and measured it against two baselines on 50 unseen demand patterns, and the result is more nuanced than the usual "RL wins" post. A single intersection still teaches most of what matters: state design, reward design and honest baselines.
Why Fixed-Time Signals Waste Your Time
Most signals run fixed-time plans. Engineers set green durations from historical traffic counts, and the plan repeats all day. Traffic does not repeat that neatly, so the plan wastes green on empty approaches while queues grow elsewhere.
Actuated signals use sensors to extend or skip phases, which helps. RL goes one step further: it learns a policy that maps current traffic conditions to a phase decision, and it optimises for long-term delay instead of reacting to one detector. The standard survey of the field collects the state, action and reward designs used across this literature, and most reported results come from simulation rather than deployed corridors [1].
How RL Frames Signal Control
Here is the setup this post builds:
- State: queue length on each approach, the current phase, and time spent in that phase.
- Action: choose the next phase, north-south green or east-west green.
- Reward: the negative total queue length at each step.
- Episode: 30 minutes of simulated traffic, one decision every 5 seconds.
Queues make a useful reward because they are simple, they correlate with delay, and the agent gets feedback at every step instead of at the end. The same formulation appears across the traffic-RL literature, usually with queue lengths or delay in the state and a queue or pressure term in the reward [1]. If you have built a constructive policy for the traveling salesman problem, the loop will feel familiar: the environment changes after every action, and the policy has to commit to a sequence.
Setting Up
pip install gymnasium stable-baselines3 numpy
The run behind this post used Python 3.12, Gymnasium 1.4.0, Stable-Baselines3 2.9.2, PyTorch 2.15.0+cpu and NumPy 2.5.3. Library versions change often, so check the docs if an import fails.
Building a Tiny Intersection Simulator
A custom environment keeps every line readable. Four approaches feed one intersection, vehicles arrive randomly, and the green phase discharges up to two cars per approach per step. Switching phases costs one step of lost time, which mimics yellow and all-red clearance.
import numpy as np
import gymnasium as gym
from gymnasium import spaces
STEP_SEC = 5
class IntersectionEnv(gym.Env):
def __init__(self, horizon=360, cap=2, max_q=40):
super().__init__()
self.horizon, self.cap, self.max_q = horizon, cap, max_q
self.action_space = spaces.Discrete(2) # 0 = NS green, 1 = EW green
self.observation_space = spaces.Box(0.0, 1.0, shape=(6,), dtype=np.float32)
def reset(self, seed=None, options=None):
super().reset(seed=seed)
ns, ew = self.np_random.uniform(0.2, 0.8, size=2)
self.rates = np.array([ns, ns, ew, ew]) # N, S, E, W arrivals per step
self.q = np.zeros(4)
self.phase, self.since, self.lost, self.t = 0, 0, 0, 0
return self._obs(), {}
def _obs(self):
return np.concatenate([
self.q / self.max_q,
[self.phase, min(self.since, 20) / 20],
]).astype(np.float32)
def step(self, action):
action = int(action)
if action != self.phase:
self.phase, self.since, self.lost = action, 0, 1
if self.lost > 0:
self.lost -= 1 # clearance step: no discharge
else:
served = [0, 1] if self.phase == 0 else [2, 3]
self.q[served] = np.maximum(0, self.q[served] - self.cap)
self.q = np.minimum(self.q + self.np_random.poisson(self.rates), self.max_q)
self.since += 1
self.t += 1
reward = -self.q.sum() / 10.0
truncated = self.t >= self.horizon
return self._obs(), reward, False, truncated, {"queue": self.q.sum()}
The lost counter is the part most toy simulators omit. Without it, the agent can flip phases every step, discharge four cars per step in both directions and beat physics. With it, every switch costs five seconds of green, so the agent has to decide whether the other queue is worth the clearance time.
Why Random Demand Matters
Each reset draws new arrival rates, so the agent cannot memorise one scenario. Rates range from 0.2 to 0.8 vehicles per approach per step, which is 144 to 576 vehicles per hour per approach, or 576 to 2,304 vehicles per hour across the four approaches. That forces a policy that works across light, heavy and lopsided traffic instead of one that overfits a single dataset.
The queue cap at 40 vehicles per approach is a simplification, not a spillback model. It keeps the state bounded and rarely binds at average demand, but it does distort the heaviest episodes.
Training a DQN Agent
The action space is tiny and discrete, so DQN is a reasonable first choice.
from stable_baselines3 import DQN
env = IntersectionEnv()
model = DQN(
"MlpPolicy", env,
learning_rate=1e-3, buffer_size=100_000, learning_starts=2_000,
batch_size=64, gamma=0.99, train_freq=4, target_update_interval=500,
exploration_fraction=0.3, exploration_final_eps=0.05,
verbose=1, seed=0,
)
model.learn(total_timesteps=150_000)
This configuration trained in 2.3 minutes on a laptop CPU, with no GPU. The hyperparameters are a small-environment adaptation of the Stable-Baselines3 DQN defaults: a shorter buffer, earlier learning starts, a smaller target-update interval and a longer exploration window for 360-step episodes [5]. Watch the mean episode reward. If it stays flat, suspect the reward or the observation before you blame DQN.
Building Two Baselines
An RL result means nothing without comparisons. This post uses a classic fixed-time plan and a simple queue-based rule.
def fixed_time(env, obs, green_steps=6):
return (env.t // green_steps) % 2
def longest_queue(env, obs, min_green=3):
if env.since < min_green:
return env.phase
return 0 if env.q[:2].sum() >= env.q[2:].sum() else 1
The fixed plan gives each direction 30 seconds of green. The queue rule serves the busier direction, but it holds each green for at least 15 seconds so it does not flip-flop. That minimum is too short: in the measured run the rule averaged 3.4-step greens and 106 switches per episode, and it paid the lost step every time.
A baseline you have not tuned is a straw man, so I swept both. Fixed-time at 20, 25, 30, 35, 40, 50 and 60 seconds scored 11.77, 9.87, 9.57, 9.55, 9.98, 10.90 and 11.98 mean queue; the best plan is 35 seconds at 9.55. The queue rule at 15, 20, 25, 30, 35 and 40 seconds scored 8.89, 7.71, 7.90, 8.45, 8.86 and 9.54; the best minimum is 20 seconds at 7.71. Tuning the rule improves it by 13%, which is exactly why comparing against the 15-second version alone would overstate the agent.
Evaluating Fairly
Test on fresh seeds the agent never saw. Each seed fixes the demand and the arrival stream, so every policy faces identical traffic.
def evaluate(policy, seeds):
results = []
for seed in seeds:
env = IntersectionEnv()
obs, _ = env.reset(seed=seed)
queues, done = [], False
while not done:
obs, _, _, done, info = env.step(policy(env, obs))
queues.append(info["queue"])
mean_q = np.mean(queues)
wait = mean_q / env.rates.sum() * STEP_SEC # Little's law estimate
results.append((mean_q, wait))
return np.mean(results, axis=0)
rl = lambda env, obs: model.predict(obs, deterministic=True)[0]
seeds = range(1000, 1050)
for name, pol in [("Fixed-time", fixed_time), ("Longest queue", longest_queue), ("DQN", rl)]:
q, w = evaluate(pol, seeds)
print(f"{name:14s} | mean queue {q:5.1f} | est. wait {w:5.1f} s")
The wait figure uses Little's law: average wait equals average queue divided by arrival rate. It is an estimate, and the queue cap distorts it under heavy load. Across the 50 seeds, an average of 684 vehicles arrived per episode and at most 0.1 were dropped by the cap, so the distortion is small on average but larger on the worst seeds. Print per-seed numbers too, because an average can hide a disaster in one scenario.
What the Numbers Actually Showed
Fifty fresh demand patterns on one four-approach intersection, same seeds for every policy, 30-minute episodes:
| Policy | Mean total queue | Est. wait | Worst seed |
|---|---|---|---|
| Fixed-time, 30 s | 9.57 | 24.1 s | 26.9 |
| Queue rule, 15 s | 8.89 | 20.9 s | 42.1 |
| Queue rule, 20 s (best) | 7.71 | 19.1 s | 25.0 |
| DQN, greedy | 5.77 | 14.4 s | 14.3 |
| DQN, sampled actions | 6.21 | 15.5 s | — |
Six things are worth more than the mean.
- The DQN beat the 30-second fixed plan by 39.7% on mean queue, and the best-tuned fixed plan (9.55) by 39.6%. It beat the 15-second queue rule on 50 of 50 seeds, a 35.1% reduction, and the best-tuned 20-second rule by 25.2%.
- The result is not seed luck. Two extra training seeds scored 5.78 and 6.17 on the same 50 evaluation seeds, beating the 15-second rule on 50/50 and 42/50 respectively.
- The DQN is also more stable. Its worst seed was 14.3, against 26.9 for fixed-time and 42.1 for the queue rule. The queue rule collapses on high-demand balanced seeds: 1029, 1031 and 1040 all have total demand between 1.37 and 1.40 vehicles per step, and the rule abandons one queue before draining it while paying the lost step.
- Fixed-time is not worst under lopsided demand in this simulator. Splitting the seeds by demand imbalance gives 9.51 for balanced and 9.63 for lopsided, with a correlation of +0.06. Its weakness is being late to every burst, not imbalance specifically, which is a useful correction to a claim repeated in many tutorials.
- The learned behaviour is not a fixed cycle in disguise. The DQN held green for about 4.0 steps on average, but 5.5 steps on the congested seed 1029, including a 7-step hold while it drained a queue from 6 to 1. The queue rule averaged 3.4 steps and switched away too early.
- The agent did not starve an approach. Mean per-approach queues were 2.87 for north-south and 2.90 for east-west, so the total-queue reward produced a balanced policy in this run.
Sampling actions at evaluation time made the DQN worse (6.21 against 5.77), so use the greedy action when you report a number. And keep the main caveat in view: the reward is the negative total queue, which is exactly the metric in the table. This is the friendliest possible arrangement for RL; a real objective such as delay, stops or emissions would make the comparison harder [1].
How to Improve Your Agent
Try these in order:
- Tune the baselines before you tune the agent. The 15-second queue rule was 13% worse than its own best setting.
- Train longer and across more seeds. The 5.77 result was stable across three seeds, but longer training may still help.
- Enrich the state with discharged vehicles, the wait of the oldest vehicle, or time of day. Queue length alone cannot distinguish a burst that is clearing from one that is building.
- Change the reward. Compare queue length, cumulative waiting time, and pressure (incoming queue minus outgoing queue), then check per-approach starvation.
- Add phases. Protected left turns turn this into a larger action space and a harder masking problem.
- Randomise demand within the episode. A rate that shifts halfway through mimics a rush hour and tests adaptation rather than a static policy.
- Try PPO or Double DQN. Algorithm choice matters less than setup, but the comparison teaches you what is stable [5].
- Move to a real simulator before you believe any of it. Start with sumo-rl [3] or CityFlow [4], and treat the toy environment as a lesson in formulation, not a traffic model.
Moving to a Real Traffic Simulator
Once the toy works, graduate to a proper simulator. SUMO is the open-source standard, and it models vehicle dynamics, lane geometry and real road networks [2]. The sumo-rl package wraps SUMO as a Gymnasium environment, so your training code barely changes [3]. CityFlow offers a faster alternative for large-scale experiments [4].
Check each project's current documentation before you commit, because installation steps and APIs change. Real simulators also expose new problems: yellow-phase rules, pedestrian phases, spillback, detector noise and slow simulation speed. The multi-intersection version adds coordination, since one signal's output becomes the next signal's input; the action-masking and baseline discipline from job shop scheduling with RL carries over, but the state space does not.
Common Pitfalls
- Training on one demand pattern. The agent memorises it and fails on anything else.
- Rewarding the wrong thing. An agent that minimises total queue may starve a quiet approach, so check per-approach waits.
- Skipping the lost-time cost. Without it, the agent flips phases constantly and cheats physics.
- Comparing against a weak baseline. Fixed-time loses to almost everything, and an untuned queue rule is not much better. Race the tuned versions.
- Forgetting that the reward is the metric. If you score mean queue and reward negative mean queue, you have made the problem easy for the agent. Report that explicitly.
- Trusting simulation blindly. A simulator captures your assumptions, not reality. Real drivers behave differently, and a learned policy needs safety layers before it touches a live signal [2].
Most failures in this pipeline come from setup and evaluation, not from the neural network.
Best Book to Read
| Book | Description | Get it |
|---|---|---|
| Traffic Engineering | the standard civil-engineering text on signal timing: cycle length, lost time, saturation flow, actuation and the delay formulas this simulator approximates. | View on Amazon |
As an Amazon Associate I earn from qualifying purchases. Links on this page are affiliate links, and I may earn a commission at no extra cost to you.
Read it after you have run the code once. It makes the difference between "my agent beats a 15-second queue rule" and "this signal would survive a real intersection" impossible to confuse.
Wrapping Up
You built a complete RL signal-control pipeline: a simulator with lost time, a DQN agent and two tuned baselines. The agent cut mean queue by 39.6% against the best fixed-time plan and 25.2% against the best queue rule, and it held up across three training seeds. It also did so while optimising the exact quantity being scored, on one abstract intersection, with no pedestrians, left turns or yellow timing.
So run the script this week, then change one thing: the reward, the demand range, or the phase count. If your agent beats the tuned queue rule, that is a real result for this simulator. If it does not, you have learned why traffic engineers are not worried about their jobs yet. For the same baseline discipline on a routing problem, see the vehicle routing comparison.
Frequently Asked Questions
Can reinforcement learning beat fixed-time traffic signals?
In this simulator it did: the trained DQN averaged a total queue of 5.77 against 9.55 for the best-tuned fixed-time plan, a 39.6% reduction, and beat the best queue-based rule by 25.2%. That is a simulation result on one intersection with no pedestrians, left turns or yellow timing, so it is a reason to test further, not a deployment claim.
Why does the longest-queue rule lose to the DQN?
It switches as soon as the other approach looks longer, so it pays a lost clearance step roughly every fourth step and abandons a queue before draining it. In this run it averaged 3.4-step greens and 106 switches per episode, against 4.0-step greens and 93 switches for the DQN. Even tuned to a 20-second minimum green, the rule reached 7.71 against the DQN's 5.77.
What reward should a traffic signal agent use?
Start with negative total queue length: it is dense, it correlates with delay, and it gives feedback every step. Be aware that it is also the metric being scored here, which flatters the agent. Try pressure (incoming queue minus outgoing queue), cumulative waiting time or throughput next, and always check per-approach queues for starvation.
How long does DQN training take for this environment?
150,000 steps took 2.3 minutes on a laptop CPU with the configuration in this post. The seed used for the main results scored 5.77, and two extra seeds scored 5.78 and 6.17, so the result is not a lucky initialization. No GPU is required for an environment this small.
Does this transfer to a real intersection?
Not directly. Move to SUMO with sumo-rl or to CityFlow, add yellow and all-red clearance, pedestrian phases, minimum and maximum greens, detector failures and a fallback timing plan. A learned policy should never have sole control of a real signal, and this simulator is too abstract to predict field performance.
Sources
[1]Hua Wei, Guanjie Zheng, Vikash Gayah and Zhenhui Li, A Survey on Traffic Signal Control Methods — the state, action and reward designs used across the field, and the simulation-heavy evidence base this tutorial follows: arxiv.org/abs/1904.08117[2]SUMO documentation — the open-source microscopic traffic simulator, including network, demand and signal-program modelling: sumo.dlr.de/docs[3]Lucas Alegre, sumo-rl — a Gymnasium wrapper for SUMO with single- and multi-agent signal-control environments: github.com/LucasAlegre/sumo-rl[4]CityFlow — a fast multi-agent traffic simulator for large-scale signal-control experiments: github.com/cityflow-project/CityFlow[5]Stable-Baselines3 DQN — the DQN implementation, hyperparameters and Double DQN option used in this post: stable-baselines3.readthedocs.io/en/master/modules/dqn.html