Contents
Figure 1: Real-Time Bidding with RL — in 100 ms, your budget competes in millions of auctions. Bid too high and you're broke by lunch; bid too low and you win nothing
A user opens a page. In about 100 milliseconds, an auction runs, dozens of advertisers bid, and one ad wins the slot. Your budget has to last all day across millions of these auctions. Bid too high and you drain it by lunch. Bid too low and you win nothing. Reinforcement learning can learn how to pace that spending, and a small simulator shows you how.
I cannot claim I have run RL bidding on a live ad platform, so treat this as a learning project. Real-time bidding (RTB) is one of the best RL sandboxes around, because the reward is clear, the feedback is fast, and the constraints are real. Ready to build?
What Is Real-Time Bidding?
Real-time bidding (RTB) is programmatic ad buying, where each ad impression gets auctioned the moment a page loads. A demand-side platform (DSP) bids on behalf of an advertiser. The DSP sees limited information about the impression, such as site category, device, and an estimated click probability, and must choose a bid price within milliseconds.
Most exchanges historically used second-price auctions, where the winner pays the second-highest bid. Many have moved toward first-price auctions, where the winner pays their own bid. That shift matters a lot for strategy. Check current exchange documentation for the auction rules that apply to you, since they change.
Why Bidding Is a Sequential Problem
Each auction looks independent, but your budget links them. Spend heavily early, and you have nothing left for the high-value evening traffic. Spend too cautiously, and you finish the day with unspent budget and missed conversions.
So the real question is not "What should I bid on this impression?" It is "How should I spread a fixed budget across thousands of impressions?" Ever blown a monthly budget in the first week? That is the trap.
Classical approaches include:
- Linear bidding: bid proportional to the estimated value of the impression, scaled by a tuned factor
- Budget pacing: throttle participation or scale bids so spend follows a target curve
- Optimal bidding theory: derive bids from the estimated win-rate curve and a Lagrangian budget constraint
Research on budget-constrained bidding, including work on model-free RL for bidding strategies, shows RL can learn adaptive scaling policies. Look up the primary papers for exact formulations and results.
How RL Frames Bidding
Here is the setup we will build:
- State: remaining budget fraction, fraction of the day elapsed, and recent win rate
- Action: choose a bid multiplier applied to each impression's estimated value
- Reward: value won (clicks, in our toy), measured per step
- Constraint: spending cannot exceed the budget
We will let the agent control a multiplier, not a raw price. This is a common design. The impression's value comes from a prediction model, and the agent decides how aggressively to bid relative to that value. It also keeps the action space small and stable.
Setting Up
pip install gymnasium stable-baselines3 numpy
Library versions change often, so check the docs if an import fails.
Building the Auction Simulator
We will simulate a day of impressions. Each impression has a true click probability, and the agent sees a noisy estimate of it. Competing bidders draw their highest bid from a distribution. We will use a second-price auction for simplicity: the agent wins if its bid beats the top competing bid, and pays that competing bid.
import numpy as np
import gymnasium as gym
from gymnasium import spaces
MULTIPLIERS = np.array([0.0, 0.2, 0.4, 0.6, 0.8, 1.0, 1.3, 1.7, 2.2, 3.0])
class BiddingEnv(gym.Env):
def __init__(self, impressions=2000, batch=10):
super().__init__()
self.n, self.batch = impressions, batch
self.horizon = impressions // batch
self.action_space = spaces.Discrete(len(MULTIPLIERS))
self.observation_space = spaces.Box(0.0, 1.0, shape=(5,), dtype=np.float32)
def reset(self, seed=None, options=None):
super().reset(seed=seed)
rng = self.np_random
self.budget0 = rng.uniform(30.0, 80.0)
self.budget = self.budget0
self.market = rng.uniform(0.6, 1.4) # competitor price level
self.t = 0
self.last_win = 0.5
self.last_spend = 0.0
self.total_clicks = 0.0
return self._obs(), {}
def _obs(self):
return np.array([
self.budget / self.budget0,
self.t / self.horizon,
self.last_win,
min(self.last_spend / (self.budget0 / self.horizon * 3), 1.0),
self.market / 1.4,
], dtype=np.float32)
def step(self, action):
mult = MULTIPLIERS[int(action)]
rng = self.np_random
# true click probability, and a noisy estimate the bidder sees
ctr = rng.beta(2, 40, self.batch)
est = np.clip(ctr * rng.lognormal(0, 0.25, self.batch), 0, 1)
value = est * 100.0 # value of an impression in $
comp = rng.lognormal(np.log(self.market * 2.0), 0.6, self.batch)
bids = mult * value
wins = (bids > comp) & (self.budget > 0)
cost = np.where(wins, comp, 0.0)
# Enforce the budget within the batch
cum = np.cumsum(cost)
affordable = cum <= self.budget
wins &= affordable
cost = np.where(wins, comp, 0.0)
spend = cost.sum()
clicks = (rng.random(self.batch) < ctr)[wins].sum()
self.budget -= spend
self.total_clicks += clicks
self.last_win = wins.mean()
self.last_spend = spend
self.t += 1
reward = float(clicks)
truncated = self.t >= self.horizon
return self._obs(), reward, False, truncated, {
"spend": spend, "clicks": clicks, "budget_left": self.budget}
Why random market conditions matter: each reset draws a new budget and a new competitor price level. The agent must learn general pacing behavior, not memorize one market. A cheap market rewards aggressive bidding, and an expensive one rewards patience.
Why the budget check matters: the simulator caps spending inside each batch. Real exchanges do not forgive overspend, so the environment makes budget exhaustion a hard constraint. An agent that empties the budget early simply stops winning, which teaches it to pace.
Training a PPO Agent
The action space is discrete, so PPO works well:
from stable_baselines3 import PPO
env = BiddingEnv()
model = PPO(
"MlpPolicy", env,
learning_rate=3e-4, n_steps=2048, batch_size=256,
gamma=0.99, gae_lambda=0.95, ent_coef=0.01,
verbose=1, seed=0,
)
model.learn(total_timesteps=300_000)
Expect a few minutes on a laptop. Watch the mean episode reward, which equals total clicks. If it stays flat, suspect your observation design before you blame PPO.
Building Baselines
An RL result means nothing in isolation. We will compare three strategies.
def make_constant(mult_index):
def policy(env, obs):
return mult_index
return policy
def make_paced(gain=3.0):
# Raise the multiplier when behind the spending curve; lower it when ahead
def policy(env, obs):
spent_frac = 1.0 - obs[0]
target = obs[1]
error = target - spent_frac # positive: underspending
desired = 1.0 + gain * error
return int(np.argmin(np.abs(MULTIPLIERS - desired)))
return policy
The constant policy bids a fixed fraction of value all day, which usually exhausts the budget early or leaves money unspent. The paced policy nudges the multiplier to track a linear spending target, a simple stand-in for production pacing controllers. Real controllers are more sophisticated, so treat this as a fair but simple baseline.
Evaluating Fairly
Test on fresh seeds. Each seed fixes the budget and the market level, so every policy faces similar conditions. The random draws inside an episode still vary, so treat comparisons as statistical.
def evaluate(policy, seeds):
rows = []
for seed in seeds:
env = BiddingEnv()
obs, _ = env.reset(seed=seed)
done = False
while not done:
obs, _, _, done, info = env.step(policy(env, obs))
spent = 1.0 - info["budget_left"] / env.budget0
rows.append((env.total_clicks, spent))
return np.mean(rows, axis=0)
rl = lambda env, obs: int(model.predict(obs, deterministic=True)[0])
seeds = range(1000, 1050)
policies = [
("Constant 0.6", make_constant(3)),
("Constant 1.0", make_constant(5)),
("Paced", make_paced()),
("PPO", rl),
]
for name, pol in policies:
clicks, spent = evaluate(pol, seeds)
print(f"{name:13s} | clicks {clicks:6.1f} | budget spent {spent:.0%}")
Report budget utilization next to clicks. A policy that wins many clicks by overspending is not possible here, but one that leaves half the budget idle is leaving value on the table. Ever celebrated a low cost-per-click on a campaign that never spent its budget?
What Results Should You Expect?
I cannot promise numbers, because they depend on your seed, training length, and market draw. In setups like this, the usual pattern looks like this:
- Constant bidding often exhausts the budget early in cheap markets or underspends in expensive ones
- The paced policy tracks the budget curve and usually beats constant bidding
- PPO can match or beat pacing by also learning to bid harder when its recent win rate looks poor, and more carefully when competition spikes
- PPO can underperform when training is short or the observation hides the market level
Also consider a fourth baseline: a tuned linear bidder where you grid-search the single best multiplier per market type. If your agent only matches that, the RL machinery did not earn its complexity. Anticlimactic? Maybe, but honest.
How to Improve Your Agent
Try these in order:
- Train longer. Double the timesteps and compare.
- Enrich the state. Add the recent average winning price and a short history of win rates.
- Add traffic variation. Make impression value follow a daily cycle, so the agent learns to save budget for good hours.
- Switch to first-price auctions. Charge the agent its own bid. Now it must learn bid shading, bidding below value to keep surplus.
- Use continuous actions. Try a continuous multiplier with SAC or PPO.
- Model conversions, not clicks. Reward delayed, sparse conversions and watch training get harder.
Where RL Meets Real Ad Systems
The toy teaches the loop. Real systems add serious complications:
- Delayed and sparse feedback: conversions arrive hours or days after the bid
- Auction format shifts: first-price auctions require bid shading, and exchange rules change
- Non-stationary competition: other bidders adapt, so your environment keeps moving
- Multi-agent effects: many learning bidders interact, which can destabilize markets
- Latency limits: the bidder must answer in milliseconds, so models stay small
- Off-policy evaluation: you cannot safely test every policy live, so teams use logged data and counterfactual methods carefully
- Privacy and policy constraints: data availability keeps changing, which limits what the state can contain
Published work, including studies using real ad logs and multi-agent formulations, gives you benchmarks and ideas. Check recent surveys on RL for computational advertising before you pick a direction, and verify tooling against current documentation.
Guardrails Before Anything Touches Production
A learned bidder should never get unchecked control of real money. Add these protections:
- Hard budget caps enforced outside the policy
- Bid ceilings so a bad action cannot overpay wildly
- Fallback to a simple pacing rule if metrics look broken
- Shadow mode first, with the agent recommending bids while the real system acts
- Spend monitoring and kill switches with automatic alerts
Treat these as non-negotiable. An RL agent that discovers a loophole in your reward can burn a budget faster than any human could.
Common Pitfalls
- Ignoring the budget constraint. The agent learns to bid high forever, then fails when real caps bite.
- Rewarding clicks only. The agent chases cheap, low-value clicks, so reward what the advertiser actually values.
- Weak baselines. Beating constant bidding proves little. Race a tuned pacing controller.
- Static simulators. Real competitors adapt, so test against shifting markets.
- Trusting offline results blindly. A simulator reflects your assumptions, not reality.
Did you notice that most of these mistakes involve an unrealistic setup, not a weak algorithm?
Best Book to Read
| Book | Description | Get it |
|---|---|---|
| Real-Time Bidding and Ad Exchanges: The Economics and Technology of Programmatic Advertising | the economics and technology of programmatic advertising, covering auction mechanics, pacing, and the measurement challenges this simulator abstracts. | View on Amazon |
As an Amazon Associate I earn from qualifying purchases. Links on this page are affiliate links, and I may earn a commission at no extra cost to you.
Read it after you have run the code once. It makes the difference between "my agent paces a beta-distributed CTR" and "this bidder would survive a real exchange" impossible to confuse.
Wrapping Up
RL fits bidding because the problem has sequential budget tradeoffs, fast feedback, and clear rewards. But a well-tuned pacing controller stays hard to beat on simplicity, safety, and explainability, and production systems keep that as a fallback. The strongest designs often combine a value-prediction model, a classical bidder, and RL for adaptive scaling.
So run the script this week, then change one thing: the auction type, the budget range, or the reward. If your agent beats the paced baseline, celebrate and test it in shadow mode. If it does not, you just learned why ad-tech teams respect their boring controllers.
Frequently Asked Questions
Can reinforcement learning beat classical budget pacing in RTB?
In this simulator it sometimes can: PPO matches or beats a proportional pacing controller by also learning to bid harder when its recent win rate drops and more carefully when competition spikes. But a well-tuned linear bidder or pacing rule is hard to beat on simplicity and safety.
Why does the agent control a bid multiplier instead of a raw price?
The impression's value comes from a prediction model. The agent scales that value, keeping the action space small and stable across impressions with different absolute values. This is a common design in production bidding systems.
What auction format does this simulator use?
Second-price: the winner pays the second-highest bid. Many exchanges have moved to first-price auctions where the winner pays their own bid, which requires bid shading. The article includes a note on switching to first-price.
How does the budget constraint work in the simulator?
Spending is capped within each batch of impressions. If cumulative cost exceeds the remaining budget, the excess wins are rejected. This mimics real exchanges that do not forgive overspend.
What guardrails are needed before using RL bidding in production?
Hard budget caps enforced outside the policy, bid ceilings, fallback to a simple pacing rule, shadow-mode evaluation, and automatic spend monitoring with kill switches. An RL agent that finds a reward loophole can burn a budget faster than any human.
Sources
[1]J. Yuan et al., Real-Time Bidding Algorithms for Performance-Based Display Ad Allocation — optimal bidding under budget constraints, linear and pacing strategies: dl.acm.org/doi/10.1145/2623330.2623357[2]W. Zhang et al., Optimal Real-Time Bidding for Display Advertising — the classic linear bidder with a budget constraint: dl.acm.org/doi/10.1145/2487575.2487663[3]K. Cai et al., Reinforcement Learning for Bid Optimization in RTB — model-free RL approaches for adaptive bidding: arxiv.org/abs/1706.08835[4]IAB Tech Lab, OpenRTB Specification — the current auction protocol and bid request/response formats: iabtechlab.com/standards/openrtb[5]Real-Time Bidding and Ad Exchanges — programmatic advertising economics and technology: Amazon search
Related Articles
- Kubernetes Autoscaling with Reinforcement Learning: Smarter Than Threshold Rules?
- Dynamic Spectrum Access with Reinforcement Learning: A Wireless Use Case
- Q-Routing Explained: Reinforcement Learning for Packet Routing in Networks
- Traffic Signal Control with Reinforcement Learning: Cut Wait Times in Simulation
- Job Shop Scheduling with Reinforcement Learning (Python Tutorial)