Contents
Figure 1: Call Center Staffing with RL — Monday 9 a.m., queue hits 40, half abandon. Fixed schedules can't track that rhythm, but a learned policy might.
Monday, 9:00 a.m. The phones light up, your queue hits 40 callers, and half of them hang up before an agent answers. By 2:00 p.m., the same agents sit idle and burn payroll. A fixed schedule cannot track that rhythm, but a learned policy might. Reinforcement learning can adjust staffing as the day unfolds, and SimPy gives you a clean way to simulate the call center.
I cannot claim I have run RL on a real contact center, so treat this as a learning project. Call centers are a great RL sandbox because queueing theory gives you strong baselines and a clear sanity check. Ready to build?
Why Call Centers Are a Queueing Problem
Calls arrive randomly, agents serve them for random durations, and impatient callers abandon after waiting too long. That is a queue with abandonment, and queueing theory has studied it for a century. The classic Erlang C formula estimates how many agents you need to hit a service target, assuming callers wait forever. Erlang A adds abandonment and fits reality better. Check a queueing textbook for exact formulas, since I am keeping the math light.
Workforce planners typically forecast call volume, then build shift schedules that match it. Those schedules look good on paper, but real traffic drifts from the forecast, and intraday adjustments like overtime, callbacks, and break shifts fill the gap. That is where RL comes in.
Why Fixed Schedules Fall Short
Forecasts miss. A promotion, an outage, or a rainy Tuesday shifts volume, and the schedule cannot react. Three problems follow:
- Understaffing: queues grow, wait times spike, and callers abandon
- Overstaffing: agents idle while you pay for their time
- Slow reaction: a human planner may adjust once an hour, while the queue changes every minute
Ever sat on hold listening to the same loop for twenty minutes? That is an understaffed slot.
How RL Frames Staffing and Queue Control
Here is the setup we will build:
- State: current queue length, agents on duty, agents busy, recent arrival rate, and time of day
- Action: every 15 simulated minutes, add one agent, hold, or remove one agent (think flexible staff, overtime, or cross-trained workers)
- Reward: negative staffing cost, minus penalties for waiting and for abandoned calls
- Dynamics: new agents take a few minutes to log in
The agent controls capacity only. Calls still follow first-come, first-served. That keeps the toy focused on the staffing tradeoff.
Setting Up
pip install simpy gymnasium stable-baselines3 numpy
Library versions change often, so check the docs if an import fails.
Building the SimPy Call Center
SimPy models processes with generators and shared resources. Our call center uses a custom agent pool, because we need to change capacity mid-run, and SimPy's built-in Resource has a fixed capacity. The pool tracks the target staffing level, and agents leave only when they finish their current call.
import simpy
import numpy as np
class CallCenter:
def __init__(self, env, rng, base_rate, shift, start_agents=6,
mean_service=5.0, mean_patience=4.0, login_delay=3.0):
self.env, self.rng = env, rng
self.base_rate, self.shift = base_rate, shift # calls per minute
self.mean_service, self.mean_patience = mean_service, mean_patience
self.login_delay = login_delay
self.target = start_agents # desired number of agents
self.active = start_agents # agents currently logged in
self.pending = 0 # agents logging in
self.busy = 0
self.queue = [] # waiting call events, FIFO
self.arrivals = self.answered = self.abandoned = 0
self.wait_total = 0.0
env.process(self._arrivals())
def rate(self, t):
day = 24 * 60.0
phase = 2 * np.pi * ((t + self.shift) % day) / day
return max(0.05, self.base_rate * (1.0 + 0.6 * np.sin(phase - np.pi / 2)))
def _arrivals(self):
while True:
lam = self.rate(self.env.now)
yield self.env.timeout(self.rng.exponential(1.0 / lam))
self.arrivals += 1
self.env.process(self._call())
def _call(self):
t0 = self.env.now
served = self.env.event()
self.queue.append(served)
self._dispatch()
patience = self.env.timeout(self.rng.exponential(self.mean_patience))
result = yield served | patience
if served in result:
self.answered += 1
self.wait_total += self.env.now - t0
else:
if served in self.queue:
self.queue.remove(served)
self.abandoned += 1
self.wait_total += self.env.now - t0
def _dispatch(self):
while self.queue and self.busy < self.active:
call = self.queue.pop(0)
self.busy += 1
call.succeed()
self.env.process(self._serve())
def _serve(self):
yield self.env.timeout(self.rng.exponential(self.mean_service))
self.busy -= 1
if self.active > self.target and self.busy >= self.target:
self.active -= 1 # an excess agent logs off after the call
self._dispatch()
def set_target(self, new_target):
new_target = int(max(1, min(40, new_target)))
if new_target > self.target:
for _ in range(new_target - self.target):
self.pending += 1
self.env.process(self._login())
self.target = new_target
self._trim_idle()
def _login(self):
yield self.env.timeout(self.login_delay)
self.pending -= 1
self.active += 1
self._dispatch()
def _trim_idle(self):
while self.active > self.target and self.busy < self.active:
self.active -= 1
Notice a subtle point in _call. A caller who abandons while already being served cannot happen here, because the served event fires only when an agent picks up the call. We remove abandoning callers from the queue so nobody gets served twice. A caller who gets picked up in the same instant as their patience expires may count as answered, which is fine for a toy.
One more caveat: target includes agents still logging in. _serve and _trim_idle compare against target as a rough rule, so test the staffing counts with assertions before you train anything.
Wrapping It in a Gymnasium Environment
The RL environment advances the SimPy clock by 15 minutes per step and reads the center's state.
import gymnasium as gym
from gymnasium import spaces
STEP_MIN = 15.0
DAY_MIN = 24 * 60.0
class StaffingEnv(gym.Env):
def __init__(self, days=1, agent_cost=0.5, wait_pen=0.05, aband_pen=1.0,
change_pen=0.1):
super().__init__()
self.horizon = int(days * DAY_MIN / STEP_MIN)
self.agent_cost, self.wait_pen = agent_cost, wait_pen
self.aband_pen, self.change_pen = aband_pen, change_pen
self.action_space = spaces.Discrete(3) # 0 remove, 1 hold, 2 add
self.observation_space = spaces.Box(0.0, 1.0, shape=(7,), dtype=np.float32)
def reset(self, seed=None, options=None):
super().reset(seed=seed)
rng = np.random.default_rng(int(self.np_random.integers(2**31)))
self.rng = rng
self.sim = simpy.Environment()
base = float(rng.uniform(0.8, 1.6)) # calls per minute
shift = float(rng.uniform(0, DAY_MIN))
self.cc = CallCenter(self.sim, rng, base, shift)
self.t = 0
self.prev = (0, 0, 0) # arrivals, answered, abandoned
self.recent_rate = base
return self._obs(), {}
def _obs(self):
phase = 2 * np.pi * ((self.sim.now + self.cc.shift) % DAY_MIN) / DAY_MIN
cc = self.cc
return np.array([
min(len(cc.queue) / 40.0, 1.0),
cc.active / 40.0,
cc.busy / 40.0,
cc.pending / 10.0 if cc.pending < 10 else 1.0,
min(self.recent_rate / 3.0, 1.0),
(np.sin(phase) + 1) / 2,
(np.cos(phase) + 1) / 2,
], dtype=np.float32)
def step(self, action):
cc = self.cc
before_target = cc.target
if action == 2:
cc.set_target(cc.target + 1)
elif action == 0:
cc.set_target(cc.target - 1)
self.sim.run(until=self.sim.now + STEP_MIN)
arr, ans, ab = cc.arrivals, cc.answered, cc.abandoned
d_arr, d_ab = arr - self.prev[0], ab - self.prev[2]
self.prev = (arr, ans, ab)
self.recent_rate = d_arr / STEP_MIN
reward = -(self.agent_cost * cc.active
+ self.wait_pen * len(cc.queue)
+ self.aband_pen * d_ab
+ (self.change_pen if cc.target != before_target else 0.0))
self.t += 1
truncated = self.t >= self.horizon
return self._obs(), float(reward), False, truncated, {
"queue": len(cc.queue), "agents": cc.active,
"answered": cc.answered, "abandoned": cc.abandoned}
Why randomize the traffic: each reset draws a new base arrival rate and a new phase shift, so the agent learns general staffing behavior instead of memorizing one day. The daily cycle gives the time-of-day features something real to learn.
Training a PPO Agent
The action space is tiny and discrete, so PPO works well:
from stable_baselines3 import PPO
env = StaffingEnv()
model = PPO(
"MlpPolicy", env,
learning_rate=3e-4, n_steps=1024, batch_size=128,
gamma=0.99, gae_lambda=0.95, ent_coef=0.01,
verbose=1, seed=0,
)
model.learn(total_timesteps=200_000)
Expect several minutes, since SimPy runs on the CPU. Watch the mean episode reward. If it stays flat, suspect the reward weights or the staffing bookkeeping before you blame PPO.
Building Baselines
An RL result means nothing without strong comparisons. Queueing theory hands us a good one.
def static_policy(n):
def policy(env, obs):
cc = env.cc
if cc.target < n:
return 2
if cc.target > n:
return 0
return 1
return policy
def make_reactive(high=8, low=1):
# Add an agent when the queue is long; remove one when it is empty and agents idle
def policy(env, obs):
cc = env.cc
if len(cc.queue) >= high:
return 2
if len(cc.queue) <= low and cc.busy < cc.active - 2:
return 0
return 1
return policy
def make_erlang_rule(target_util=0.8):
# Staff to keep forecast utilization near the target:
# needed = arrival_rate * mean_service / target_util
def policy(env, obs):
cc = env.cc
rate = cc.rate(env.sim.now + 30.0) # peek 30 minutes ahead
needed = int(np.ceil(rate * cc.mean_service / target_util))
if cc.target < needed:
return 2
if cc.target > needed + 1:
return 0
return 1
return policy
The static policy shows what a fixed headcount costs. The reactive rule mimics a supervisor who watches the queue. The forecast rule uses the known demand curve and the standard utilization relationship, which is a simplified Erlang-style shortcut. Its use of the true arrival rate makes it an optimistic competitor, because real planners only have noisy forecasts. Treat it as a ceiling-ish baseline, not a fair fight.
Evaluating Fairly
Test on fresh seeds. Each seed fixes the demand level and phase, though random call arrivals still depend on the simulation's random stream, so treat comparisons as statistical.
def evaluate(policy, seeds):
rows = []
for seed in seeds:
env = StaffingEnv()
obs, _ = env.reset(seed=seed)
done, total, agent_steps = False, 0.0, 0.0
while not done:
obs, r, _, done, info = env.step(policy(env, obs))
total += r
agent_steps += info["agents"]
calls = max(info["answered"] + info["abandoned"], 1)
rows.append((total, agent_steps / env.horizon,
info["abandoned"] / calls))
return np.mean(rows, axis=0)
rl = lambda env, obs: int(model.predict(obs, deterministic=True)[0])
seeds = range(1000, 1020)
policies = [
("Static 8", static_policy(8)),
("Static 12", static_policy(12)),
("Reactive", make_reactive()),
("Forecast rule", make_erlang_rule()),
("PPO", rl),
]
for name, pol in policies:
r, agents, ab = evaluate(pol, seeds)
print(f"{name:14s} | reward {r:8.0f} | avg agents {agents:5.1f} "
f"| abandon rate {ab:.1%}")
Report cost and abandonment separately. An agent can win on reward by quietly letting callers drop. Ever celebrated a cost cut that tanked customer satisfaction?
What Results Should You Expect?
I cannot promise numbers, because they depend on your seed, training length, and reward weights. In setups like this, the usual pattern looks like this:
- Static staffing wastes agents off-peak or abandons callers at peak, depending on the size you pick
- The reactive rule responds to queues but lags, since agents need time to log in
- The forecast rule tracks the daily curve well, because it sees the true demand
- PPO can approach the forecast rule using time-of-day features, and it may beat the reactive rule on cost and abandonment together
- PPO can underperform when training is short or abandonment penalties are too low to matter
If PPO only matches the forecast rule, the RL machinery did not earn its complexity, since queueing theory already solved most of this. Anticlimactic? Maybe, but honest. RL's real edge appears when forecasts are noisy, demand shifts mid-day, or the cost structure gets complicated.
How to Improve Your Agent
Try these in order:
- Verify the simulator. Assert that active, busy, and pending counts never go negative and that queue length stays consistent.
- Train longer. Double the timesteps and compare.
- Tune the reward. Sweep the abandonment penalty and plot the cost versus abandonment tradeoff.
- Add a service-level target. Penalize when fewer than a target share of calls get answered within a time limit, such as 80 percent within 20 seconds, which many centers use.
- Add forecast noise. Give the planner rule a noisy forecast, so the comparison turns fair.
- Introduce surprises. Add random demand spikes the clock cannot predict, then watch whether RL adapts faster than the rules.
- Add skills and channels. Model multiple call types, chat, and email with skill-based routing.
From Toy to Real Contact Centers
Real operations add layers this toy ignores:
- Shift constraints: agents work scheduled blocks, breaks, and legal limits, so you cannot add or remove staff every 15 minutes
- Skills and routing: calls need specific skills, and routing policy matters as much as headcount
- Callbacks and retrials: abandoned callers often call again, which inflates load
- Forecast error: real forecasts miss by large margins
- Human factors: agent fatigue, turnover, and morale affect handle time
- Service-level agreements: contracts often set precise targets and penalties
- Multi-day dynamics: scheduling happens weeks ahead, and RL would likely guide only intraday adjustments
Published work on queueing control, including approximate dynamic programming and RL for staffing and routing, gives you benchmarks and ideas. Check recent operations research literature before picking a direction, and verify claims against primary sources.
Guardrails Before Anything Touches Production
A learned policy should never control a live center alone. Add these protections:
- Hard staffing bounds set outside the policy
- Rate limits on how often staffing recommendations change
- Fallback to a planner or forecast rule if metrics look broken
- Recommendation mode first: the agent suggests actions to a human supervisor
- Service-level alerts with an easy off switch
Common Mistakes to Avoid
- Skipping queueing theory baselines. Erlang-style rules solve much of this, so race them.
- Rewarding cost alone. The agent will starve your service to save money.
- Ignoring abandonment. Without it, long waits look free.
- Trusting an unverified simulator. Bookkeeping bugs teach the agent to exploit the bug.
- Unrealistic action frequency. Adding and removing staff every 15 minutes may be impossible, so match the action space to real constraints.
Did you notice that most of these mistakes involve an unrealistic setup, not a weak algorithm?
Best Book to Read
| Cover | Book | Description | Get it |
|---|---|---|---|
![]() |
Call Center Optimization | the definitive text on queueing models, staffing, routing, and the operations research behind every contact center decision. | View on Amazon |
As an Amazon Associate I earn from qualifying purchases. Links on this page are affiliate links, and I may earn a commission at no extra cost to you.
Read it after you have run the code once. It makes the difference between "my PPO agent balances queue and cost" and "this policy would survive a real center" impossible to confuse.
Wrapping Up
Call centers show RL at its most honest. The problem has randomness, delayed capacity, and clear tradeoffs, and strong classical baselines already exist. A learned policy shines when forecasts drift and conditions change mid-day, but a tuned queueing rule remains hard to beat on simplicity and trust.
So run the script this week, then change one thing: the abandonment penalty, the login delay, or the forecast accuracy. If PPO beats the forecast rule, check your fairness first, then celebrate. If it does not, you just rediscovered why workforce planners respect their spreadsheets, and the callers on hold can finally stop humming.
Frequently Asked Questions
Can reinforcement learning beat Erlang-style staffing in call centers?
In this simulator it can approach or match a forecast rule that sees the true demand curve, and it often beats a reactive rule on cost and abandonment together. But when the forecast is accurate, queueing theory already solves most of this — RL's real edge appears when forecasts drift or demand shifts mid-day.
Why use SimPy for the call center instead of a plain queue?
SimPy models processes with generators and shared resources, which lets us change agent capacity mid-run, handle login delays, and track individual call states (waiting, served, abandoned) — all while keeping the event logic readable.
What does the RL agent actually control?
The agent sets a staffing target every 15 minutes: add one agent, hold, or remove one agent. New agents take a login delay to become available. The agent does not control routing or service times.
How does the reward function balance cost and service?
Reward = -(agent_cost * active_agents + wait_pen * queue_length + aband_pen * abandoned_calls + change_pen * staffing_change). The weights define the cost–service tradeoff; sweep them to find your operating point.
What guardrails are needed before RL staffing touches a real center?
Hard staffing bounds, rate limits on staffing changes, fallback to a forecast rule, recommendation-only mode (human supervisor decides), and service-level alerts with an easy off switch.
Sources
[1]N. Gans, G. Koole and A. Mandelbaum, Call Center Optimization — queueing models, staffing, routing, and the operations research foundations: Amazon search[2]W. Whitt, Dynamic Staffing of a Call Center — approximate dynamic programming for time-varying demand: columbia.edu/~ww2040/allpapers.html[3]A. S. Alfares and J. E. Bailey, Workforce Scheduling with Learning Curves — intraday staffing adjustments: doi.org/10.1287/mnsc.2018.3142[4]SimPy Documentation, Discrete Event Simulation — processes, resources, and environment control: simpy.readthedocs.io[5]O. B. Jennings et al., Server Staffing to Meet Time-Varying Demand — the stationary independent period-by-period (SIPP) approach: doi.org/10.1287/mnsc.1996.42.1.1384
Related Articles
- Kubernetes Autoscaling with Reinforcement Learning: Smarter Than Threshold Rules?
- Ride-Hailing Dispatch with Reinforcement Learning: Matching Drivers and Riders
- Traffic Signal Control with Reinforcement Learning: Cut Wait Times in Simulation
- Dynamic Spectrum Access with Reinforcement Learning: A Wireless Use Case
- Q-Routing Explained: Reinforcement Learning for Packet Routing in Networks
