Sam Austinon November 5, 202616 min read

Kubernetes Autoscaling with Reinforcement Learning: Smarter Than Threshold Rules?

Kubernetes Autoscaling with Reinforcement Learning: Smarter Than Threshold Rules?
Contents

Figure 1: Kubernetes Autoscaling with RL — your traffic doubles at 9 a.m., but the threshold rule reacts at 9:03. RL can learn to scale early

Your traffic doubles at 9:00 a.m. sharp. The Horizontal Pod Autoscaler notices at 9:01, launches pods at 9:02, and those pods become ready at 9:03. Meanwhile your users stare at spinners. Can reinforcement learning learn to scale before the spike hits, instead of reacting after it?

I cannot claim I have run RL autoscaling on a production cluster, so treat this as a learning project. The honest answer is "sometimes, and only with strong guardrails." We will build a small simulator, train an agent, and race it against a threshold rule. Ready?

How Threshold Autoscaling Works

The standard Horizontal Pod Autoscaler (HPA) compares a metric, usually CPU utilization, against a target. It computes desired replicas with a simple proportional formula: scale the current replica count by the ratio of observed metric to target. It also applies stabilization windows to avoid flapping. Check the current Kubernetes documentation for exact behavior, because defaults and options change between versions.

Threshold rules are popular for good reasons:

  • Predictable: you can explain every decision
  • Simple: one metric and one target cover many services
  • Battle-tested: teams run them in production every day

They also have weaknesses. They react instead of anticipate, and pod startup delay means capacity arrives late. A single CPU target also ignores queue depth, request latency, and cost. Ever tuned a threshold for one traffic pattern and watched it fail on the next?

Why RL Might Help

Autoscaling is a sequential decision problem under uncertainty. Scaling up now costs money immediately and pays off only after startup delay. Scaling down saves money but risks a latency spike if demand returns.

An RL agent can learn these tradeoffs from experience:

  • Anticipation: it can learn daily and weekly patterns and scale early
  • Delay awareness: it learns that new pods take time to become ready
  • Multi-objective tradeoffs: one reward can balance latency, cost, and stability
  • Adaptation: it can keep learning as traffic changes

None of this guarantees a win. A tuned predictive rule can capture much of the same benefit, so keep that comparison in mind.

How RL Frames Autoscaling

Here is the setup we will build:

  1. State: current load, ready replicas, pending replicas, queue length, and time of day
  2. Action: remove one replica, keep the count, or add one replica
  3. Reward: negative cost of running replicas, minus a penalty for queued (unserved) requests, minus a small penalty for scaling actions
  4. Dynamics: new replicas take several steps to become ready

Including time of day gives the agent a chance to learn daily patterns. Including pending replicas lets it account for startup delay.

Setting Up

pip install gymnasium stable-baselines3 numpy

Library versions change often, so check the docs if an import fails.

Building the Autoscaling Simulator

Each step represents 30 seconds. Traffic follows a daily cycle with random bursts and noise. Each ready replica serves a fixed number of requests per step. Unserved requests join a queue.

import numpy as np
import gymnasium as gym
from gymnasium import spaces

STEPS_PER_DAY = 2880          # 30-second steps
CAPACITY = 50.0               # requests one replica serves per step
STARTUP = 4                   # steps before a new replica is ready
MAX_REPLICAS = 30

class AutoscaleEnv(gym.Env):
    def __init__(self, horizon=2 * STEPS_PER_DAY, cost=0.4,
                 queue_pen=0.02, change_pen=0.1):
        super().__init__()
        self.horizon = horizon
        self.cost, self.queue_pen, self.change_pen = cost, queue_pen, change_pen
        self.action_space = spaces.Discrete(3)   # 0 down, 1 hold, 2 up
        self.observation_space = spaces.Box(
            -1.0, 1.0, shape=(8,), dtype=np.float32)

    def _demand(self):
        phase = 2 * np.pi * ((self.t + self.shift) % STEPS_PER_DAY) / STEPS_PER_DAY
        base = self.base * (1.0 + 0.6 * np.sin(phase - np.pi / 2))
        if self.burst_left > 0:
            self.burst_left -= 1
            base *= self.burst_mult
        elif self.np_random.random() < 0.002:
            self.burst_left = int(self.np_random.integers(10, 40))
            self.burst_mult = self.np_random.uniform(1.5, 2.5)
        return max(0.0, base * self.np_random.normal(1.0, 0.08))

    def reset(self, seed=None, options=None):
        super().reset(seed=seed)
        self.base = self.np_random.uniform(300, 700)
        self.shift = int(self.np_random.integers(STEPS_PER_DAY))
        self.t, self.queue = 0, 0.0
        self.ready = 8
        self.pending = []            # steps remaining for each starting replica
        self.burst_left, self.burst_mult = 0, 1.0
        self.last_load = self.base
        self.last_load2 = self.base
        return self._obs(), {}

    def _obs(self):
        phase = 2 * np.pi * ((self.t + self.shift) % STEPS_PER_DAY) / STEPS_PER_DAY
        scale = 1000.0
        return np.array([
            min(self.last_load / scale, 1.0),
            min((self.last_load - self.last_load2) / scale + 0.5, 1.0) * 2 - 1,
            self.ready / MAX_REPLICAS,
            len(self.pending) / MAX_REPLICAS,
            min(self.queue / scale, 1.0),
            np.sin(phase), np.cos(phase),
            min(self.ready * CAPACITY / max(self.last_load, 1.0), 2.0) - 1,
        ], dtype=np.float32)

    def step(self, action):
        action = int(action)
        total = self.ready + len(self.pending)
        if action == 2 and total < MAX_REPLICAS:
            self.pending.append(STARTUP)
        elif action == 0 and self.ready > 1:
            self.ready -= 1

        # Pending replicas count down; finished ones become ready
        self.pending = [p - 1 for p in self.pending]
        self.ready += sum(1 for p in self.pending if p <= 0)
        self.pending = [p for p in self.pending if p > 0]

        demand = self._demand()
        arrivals = demand + self.queue
        served = min(arrivals, self.ready * CAPACITY)
        self.queue = min(arrivals - served, 5000.0)

        reward = -(self.cost * (self.ready + len(self.pending))
                   + self.queue_pen * self.queue
                   + (self.change_pen if action != 1 else 0.0))

        self.last_load2, self.last_load = self.last_load, demand
        self.t += 1
        truncated = self.t >= self.horizon
        return self._obs(), reward, False, truncated, {
            "queue": self.queue, "replicas": self.ready + len(self.pending)}

Why randomize the traffic: each reset draws a new base load and a new phase shift. The agent must therefore learn general scaling behavior, not memorize one day. Real traffic also bursts, so the simulator injects random spikes the agent cannot predict from the clock alone.

Training a PPO Agent

The action space is small and discrete, so either DQN or PPO works. I will use PPO:

from stable_baselines3 import PPO

env = AutoscaleEnv()
model = PPO(
    "MlpPolicy", env,
    learning_rate=3e-4, n_steps=2048, batch_size=256,
    gamma=0.995, gae_lambda=0.95, ent_coef=0.01,
    verbose=1, seed=0,
)
model.learn(total_timesteps=400_000)

Expect several minutes on a laptop. Watch the mean episode reward. If it stays flat, suspect your reward scale or observation design before you blame PPO.

Building Baselines

An RL result means nothing in isolation. We will use three comparisons.

def static_policy(replicas):
    def policy(env, obs):
        total = env.ready + len(env.pending)
        if total < replicas:
            return 2
        if total > replicas:
            return 0
        return 1
    return policy

def make_hpa(target=0.6, tolerance=0.1, cooldown=10):
    state = {"last_change": -999}
    def policy(env, obs):
        if env.t == 0:
            state["last_change"] = -999
        util = env.last_load / max(env.ready * CAPACITY, 1.0)
        total = env.ready + len(env.pending)
        if env.t - state["last_change"] < cooldown:
            return 1
        if util > target + tolerance and total < MAX_REPLICAS:
            state["last_change"] = env.t
            return 2
        if util < target - tolerance and env.ready > 1:
            state["last_change"] = env.t
            return 0
        return 1
    return policy

The static policy holds a fixed replica count, which shows what overprovisioning costs. The HPA-style rule mimics threshold scaling with a tolerance band and cooldown. It is a simplification, since the real HPA scales by a proportional formula and can add several pods at once, so treat it as a fair stand-in, not a replica of Kubernetes behavior.

Evaluating Fairly

Test on fresh seeds the agent never saw. Each seed fixes the base load, phase, and starting state, so every policy faces similar conditions.

def evaluate(policy, seeds):
    rows = []
    for seed in seeds:
        env = AutoscaleEnv()
        obs, _ = env.reset(seed=seed)
        reps, queues, rewards, bad, done = [], [], 0.0, 0, False
        while not done:
            obs, r, _, done, info = env.step(policy(env, obs))
            rewards += r
            reps.append(info["replicas"])
            queues.append(info["queue"])
            bad += info["queue"] > 100       # steps with a noticeable backlog
        rows.append((rewards, np.mean(reps), np.mean(queues), bad / env.horizon))
    return np.mean(rows, axis=0)

rl = lambda env, obs: int(model.predict(obs, deterministic=True)[0])
seeds = range(1000, 1030)
policies = [
    ("Static 12", static_policy(12)),
    ("HPA-style", make_hpa()),
    ("PPO", rl),
]
for name, pol in policies:
    r, rep, q, bad = evaluate(pol, seeds)
    print(f"{name:10s} | reward {r:9.0f} | avg replicas {rep:5.1f} "
          f"| avg queue {q:6.1f} | backlog steps {bad:.1%}")

Do not judge by reward alone. Report cost and backlog separately, because an agent can win on reward by quietly trading reliability for savings. Ever celebrated a cost cut that triggered an outage?

What Results Should You Expect?

I cannot promise numbers, because they depend on your seed, training length, and reward weights. In setups like this, the usual pattern looks like this:

  • Static provisioning wastes money off-peak, or drops requests at peak if you under-size it
  • The HPA-style rule tracks the daily cycle but lags sharp bursts because of startup delay
  • PPO can anticipate the daily ramp using the time-of-day features, and it often trims both cost and backlog
  • PPO can also underperform when training is short or the reward weights tempt it to under-provision

Also consider a fourth baseline: a tuned predictive rule that scales up a fixed number of steps before the known daily peak. If your agent only matches that rule, the RL machinery did not earn its complexity. Is that an anticlimax? Maybe, but it is an honest result.

How to Improve Your Agent

Try these in order:

  1. Train longer. Double the timesteps and compare.
  2. Tune the reward weights. The cost, queue, and change penalties define the behavior. Sweep them and plot the cost versus backlog tradeoff.
  3. Add an SLO-style constraint. Penalize heavily when backlog exceeds a threshold, mimicking a latency objective.
  4. Let actions be larger. Allow adding or removing several replicas at once for faster response.
  5. Use a recurrent policy. Try the LSTM variant in sb3-contrib so the agent can infer hidden burst state.
  6. Stress-test. Evaluate on traffic shapes the agent never trained on, such as a flash crowd or a flat load.

Guardrails Before Anything Touches Production

A learned policy should never get sole control of a live cluster. Add these protections:

  • Hard replica bounds: enforce minimum and maximum counts outside the policy
  • Rate limits: cap how many replicas change per minute
  • Fallback to HPA: switch to a threshold rule if metrics go missing or the policy misbehaves
  • Shadow mode first: let the agent recommend actions while the real autoscaler acts, and compare offline
  • Rollback and alerting: monitor SLOs and make disabling the agent a one-step operation

Real clusters also add complications this toy ignores: node provisioning delays from cluster autoscalers, pod disruption budgets, multi-service dependencies, noisy metrics, and cost models with spot and reserved capacity.

Tools Worth Knowing

  • Kubernetes HPA and VPA: your baseline and an alternative axis, since vertical scaling changes the problem
  • KEDA: event-driven autoscaling on queue length and other external metrics
  • Prometheus: the usual source of the metrics your agent would observe
  • Gymnasium and Stable-Baselines3: for building and training the agent
  • Simulation traces: replay real traffic traces in your environment before trusting a policy

Check each project's current documentation before you commit, because APIs and defaults change. Also look for recent papers and surveys on RL-based autoscaling and cloud resource management, and verify claims against the primary sources.

Common Pitfalls

  1. Training on one traffic pattern. The agent memorizes it and fails on anything else.
  2. Ignoring startup delay. If pods appear instantly in your simulator, the agent learns an easier world.
  3. Rewarding cost alone. The agent will starve your service to save money.
  4. Weak baselines. Beating a static count proves nothing. Race a tuned HPA rule and a predictive rule.
  5. Trusting simulation blindly. Synthetic traffic hides real-world messiness, so replay production traces.

Did you notice that most of these mistakes involve an unrealistic setup, not a weak algorithm?

Best Book to Read

CoverBookDescriptionGet it
Cover of “Kubernetes in Action” Kubernetes in Actionby Marko Lukša the Kubernetes foundation you need before HPA, VPA, KEDA, and cluster autoscaling make sense rather than feeling like magic. View on Amazon

As an Amazon Associate I earn from qualifying purchases. Links on this page are affiliate links, and I may earn a commission at no extra cost to you.

Read it after you have run the code once. It makes the difference between "my agent scales on a sine wave" and "this policy would survive a real cluster" impossible to confuse.

Wrapping Up

Can RL beat threshold rules? Sometimes, mainly when traffic has predictable structure, startup delays matter, and the reward captures what you care about. A tuned threshold or predictive rule stays hard to beat on simplicity, explainability, and operational safety, and the best production designs keep it as a fallback.

So run the script this week, then change one thing: the reward weights, the startup delay, or the traffic shape. If your agent beats the tuned rule, celebrate and shadow-test it. If it does not, you just saved your on-call team from a very exciting pager night.

Frequently Asked Questions

Can reinforcement learning beat the Kubernetes Horizontal Pod Autoscaler?

In this simulator it sometimes can: PPO with time-of-day features anticipates the daily ramp and trims both cost and backlog compared to a threshold rule that lags behind startup delay. But a tuned predictive rule that scales up before the known peak often matches RL on simplicity, so the complexity must earn its keep.

Why include time of day in the state?

It gives the agent a chance to learn daily and weekly patterns. Without it, the agent can only react to current load, not anticipate predictable spikes like the 9 a.m. traffic surge.

What reward function works for autoscaling?

A composite: negative cost of running replicas, minus a queue backlog penalty (unserved requests), minus a small scaling-action penalty to discourage flapping. The weights define the cost-reliability tradeoff; sweep them to find your operating point.

Does RL autoscaling need guardrails before production?

Absolutely. Hard replica bounds, rate limits on scaling actions, fallback to HPA, shadow-mode evaluation, and one-step rollback are minimum protections. A learned policy should never have sole control of a live cluster.

How does this transfer to a real Kubernetes cluster?

Not directly. Real clusters add node provisioning delays, pod disruption budgets, multi-service dependencies, noisy metrics, and spot/reserved capacity pricing. Start by replaying production traffic traces in the simulator, then shadow-test recommendations against the real HPA before any live control.

Sources

  • [3] KEDA Project, Event-driven Autoscaling — scaling on queue length, Kafka lag, Prometheus metrics, and more: keda.sh
  • [4] C. Xu et al., Reinforcement Learning for Kubernetes Autoscaling: A Survey — RL approaches for cloud resource management and autoscaling: arxiv.org/abs/2203.11218
  • [5] Marko Lukša, Kubernetes in Action — HPA, VPA, cluster autoscaler, and the production operations context: Amazon search

What are You Looking For?

esc