Contents
Figure 1: Dynamic Spectrum Access with Reinforcement Learning — twelve channels, one radio, and the race to find idle airtime
Your radio wants to send a packet, but twelve channels look equally unpromising. Some belong to licensed users who get priority, and others sit idle for no obvious reason. Guess wrong and you collide; guess right and you steal free airtime. Reinforcement learning can learn which channels to trust, and a small simulation shows you how.
I cannot claim I have deployed this on real radio hardware, so treat this as a learning project. Spectrum access is a perfect RL sandbox, because it forces you to balance exploring unknown channels against exploiting known good ones. Ready to build?
What Is Dynamic Spectrum Access?
Radio spectrum is scarce, and fixed licensing leaves much of it idle at any given moment. Dynamic spectrum access (DSA) lets a secondary user (SU) borrow channels that the primary user (PU) is not using. The SU must sense or predict channel occupancy and avoid interfering with the PU.
That makes DSA a decision problem under uncertainty. The SU never sees the full picture, and the PU's behavior follows hidden patterns. Cognitive radio is the broader field that studies these ideas. Check current regulatory frameworks, such as shared-spectrum systems like CBRS in the US, before you assume how real deployments work.
Why RL Fits This Problem
Classical approaches use fixed sensing schedules or hand-built statistical models of PU activity. Those break when traffic patterns shift. RL learns directly from outcomes, so it adapts without an explicit traffic model.
Here are the properties that make RL attractive:
- Unknown dynamics: the agent does not need the PU's occupancy model in advance
- Sequential decisions: today's channel choice informs tomorrow's
- Non-stationarity: the agent can keep learning as patterns drift
- Simple feedback: a transmission either succeeds or collides
Ever picked a checkout lane at the grocery store based on last week's experience? That is the exploration-exploitation tradeoff in daily life.
How RL Frames Spectrum Access
Here is the setup we will build:
- State: the observed occupancy history of the channels, such as the last few sensing results
- Action: choose which channel to sense and transmit on this time slot
- Reward: +1 for a successful transmission, and a penalty for a collision with the PU
- Environment: each channel follows a two-state Markov chain, busy or idle
The two-state Markov model is the standard starting point in the literature. Each channel flips between busy and idle with fixed probabilities, and different channels get different probabilities. That gives the agent real structure to learn.
Setting Up
pip install gymnasium stable-baselines3 numpy
Library versions change often, so check the docs if an import fails.
Building the Spectrum Environment
The agent picks one channel per slot. It observes whether that channel was busy, but it only sees the channel it chose, which makes this a partially observable problem. We feed the agent a short history of its own observations to compensate.
import numpy as np
import gymnasium as gym
from gymnasium import spaces
class SpectrumEnv(gym.Env):
def __init__(self, n_channels=12, history=4, horizon=500, collision_pen=1.0):
super().__init__()
self.n, self.h, self.horizon, self.pen = n_channels, history, horizon, collision_pen
self.action_space = spaces.Discrete(n_channels)
# per history slot: one-hot chosen channel + observed busy flag
self.observation_space = spaces.Box(
0.0, 1.0, shape=(history * (n_channels + 1),), dtype=np.float32)
def reset(self, seed=None, options=None):
super().reset(seed=seed)
# Each channel: P(idle -> busy) and P(busy -> idle)
self.p_ib = self.np_random.uniform(0.05, 0.6, self.n)
self.p_bi = self.np_random.uniform(0.05, 0.6, self.n)
stationary_busy = self.p_ib / (self.p_ib + self.p_bi)
self.busy = self.np_random.random(self.n) < stationary_busy
self.hist = np.zeros((self.h, self.n + 1), dtype=np.float32)
self.t = 0
return self._obs(), {}
def _obs(self):
return self.hist.flatten()
def _advance(self):
flip_to_busy = (~self.busy) & (self.np_random.random(self.n) < self.p_ib)
flip_to_idle = self.busy & (self.np_random.random(self.n) < self.p_bi)
self.busy = (self.busy | flip_to_busy) & ~flip_to_idle
def step(self, action):
a = int(action)
collided = bool(self.busy[a])
reward = -self.pen if collided else 1.0
record = np.zeros(self.n + 1, dtype=np.float32)
record[a] = 1.0
record[-1] = float(collided)
self.hist = np.vstack([self.hist[1:], record])
self._advance()
self.t += 1
truncated = self.t >= self.horizon
return self._obs(), reward, False, truncated, {"success": not collided}
Why random channel statistics matter: each reset draws new transition probabilities. The agent must therefore learn how to learn, inferring which channels look reliable from its recent history. If you fix the statistics, a simple bandit with memory would suffice, and the agent could memorize the best channel.
Training a DQN Agent
The action space is small and discrete, so DQN is a reasonable first choice.
from stable_baselines3 import DQN
env = SpectrumEnv()
model = DQN(
"MlpPolicy", env,
learning_rate=5e-4, buffer_size=100_000, learning_starts=2_000,
batch_size=64, gamma=0.9, train_freq=4, target_update_interval=500,
exploration_fraction=0.3, exploration_final_eps=0.05,
verbose=1, seed=0,
)
model.learn(total_timesteps=200_000)
Expect a few minutes on a laptop. Watch the mean episode reward. If it hovers near the random baseline, suspect your observation design first.
Building Baselines
An RL result means nothing in isolation. We will compare against three simple strategies.
def random_policy(env, obs):
return int(env.np_random.integers(env.n))
def make_epsilon_greedy(eps=0.1):
# Tracks a running success estimate per channel, resetting each episode
state = {"counts": None, "succ": None, "last_t": -1}
def policy(env, obs):
if env.t <= state["last_t"] or state["counts"] is None:
state["counts"] = np.ones(env.n)
state["succ"] = np.ones(env.n)
state["last_t"] = env.t
if env.np_random.random() < eps:
return int(env.np_random.integers(env.n))
return int(np.argmax(state["succ"] / state["counts"]))
return policy, state
def oracle_policy(env, obs):
# Cheats: sees the true channel states. Upper bound for a one-step view.
idle = np.flatnonzero(~env.busy)
return int(idle[0]) if len(idle) else 0
The epsilon-greedy policy needs the outcome fed back after each step, so we update its state inside the evaluation loop. The oracle is not a fair competitor — it shows the ceiling for any one-step decision, which tells you how much room remains.
Evaluating Fairly
Test on fresh seeds. Each seed fixes the channel statistics and the starting state, so every policy faces similar conditions. Note that the random draws inside an episode depend on each policy's actions, so treat comparisons as statistical, not exact.
def evaluate(policy, seeds, bandit_state=None):
scores, succ_rates = [], []
for seed in seeds:
env = SpectrumEnv()
obs, _ = env.reset(seed=seed)
if bandit_state is not None:
bandit_state["counts"] = None
total, ok, done = 0.0, 0, False
while not done:
a = policy(env, obs)
obs, r, _, done, info = env.step(a)
total += r
ok += info["success"]
if bandit_state is not None:
bandit_state["counts"][a] += 1
bandit_state["succ"][a] += float(info["success"])
scores.append(total)
succ_rates.append(ok / env.horizon)
return np.mean(scores), np.mean(succ_rates)
rl = lambda env, obs: model.predict(obs, deterministic=True)[0]
eps_policy, eps_state = make_epsilon_greedy(0.1)
seeds = range(1000, 1050)
rows = [
("Random", evaluate(random_policy, seeds)),
("Eps-greedy", evaluate(eps_policy, seeds, eps_state)),
("DQN", evaluate(rl, seeds)),
("Oracle", evaluate(oracle_policy, seeds)),
]
for name, (score, rate) in rows:
print(f"{name:11s} | reward {score:7.1f} | success rate {rate:.2%}")
Print per-seed numbers too. Ever trusted an average that hid one disastrous scenario?
What Results Should You Expect?
I cannot promise numbers, because they depend on your seed, training length, and the channel statistics you draw. In setups like this, the usual pattern looks like this:
- Random performs worst, and its success rate roughly matches the average idle fraction
- Epsilon-greedy is a strong competitor, because it learns each channel's long-run reliability
- DQN with history can edge ahead when it exploits short-term channel memory, since a channel that was just idle tends to stay idle
- The oracle sets the ceiling, and no learned policy should match it
If DQN loses to epsilon-greedy, that is a fair result. It usually means training stopped early or the history window is too short.
How to Improve Your Agent
Try these in order:
- Train longer. Double the timesteps and compare.
- Lengthen the history. Try 8 or 16 slots, or switch to a recurrent policy such as the LSTM variant in sb3-contrib.
- Tune the collision penalty. Higher penalties make the agent protect the PU more aggressively.
- Add sensing cost. Charge energy for sensing, and let the agent choose between sensing and transmitting blindly.
- Let the agent sense multiple channels. Real radios often sense a subset before transmitting.
- Make the PU adversarial or time-varying. Change the transition probabilities mid-episode to test adaptation.
Where RL Meets Real Wireless Systems
The toy teaches you the loop. Real systems add complications:
- Multiple secondary users compete for the same channels, so you need multi-agent RL and coordination or contention rules
- Sensing errors produce false alarms and missed detections, which turns the observation into a noisy signal
- Hard interference limits apply, so a PU collision may carry regulatory consequences, not just a reward penalty
- Hardware latency limits how fast the agent can switch channels
- Wi-Fi and 5G coexistence adds unlicensed-band problems like LTE-U and NR-U sharing with Wi-Fi
Published work on DQN and actor-critic methods for spectrum access, including multi-user settings, gives you benchmarks to compare against. Check recent surveys on cognitive radio and deep RL for spectrum sharing before you pick a direction, and verify tooling like ns-3 or GNU Radio against current documentation if you want a higher-fidelity simulator.
Common Pitfalls
- Training on fixed channel statistics. The agent memorizes the best channel and fails when patterns change.
- Ignoring collision cost. A reward that only counts throughput teaches the agent to be a bad neighbor.
- Weak baselines. Random loses to everything. Race epsilon-greedy and report the oracle gap.
- Skipping partial observability. If the agent sees every channel, you have built an easier problem than the real one.
- Trusting simulation blindly. Markov channels are a clean abstraction, and real PU traffic is burstier and more correlated.
Did you notice that most of these mistakes involve an unrealistic setup?
Best Book to Read
| Cover | Book | Description | Get it |
|---|---|---|---|
![]() |
Cognitive Radio Communications and Networks | the standard text on spectrum sensing, dynamic access and the regulatory landscape this simulator abstracts. | View on Amazon |
As an Amazon Associate I earn from qualifying purchases. Links on this page are affiliate links, and I may earn a commission at no extra cost to you.
Read it after you have run the code once. It makes the difference between "my agent beats epsilon-greedy on Markov channels" and "this policy would survive a real radio" impossible to confuse.
Wrapping Up
You built a complete RL spectrum-access pipeline: a partially observable channel simulator, a DQN agent with observation history, and honest baselines including an oracle ceiling. The code stays small, but the ideas scale to real cognitive radio research. A realistic observation model, a collision-aware reward, and strong baselines matter more than any fancy network.
So run the script this week, then change one thing: the history length, the collision penalty, or the number of channels. If your agent beats epsilon-greedy, celebrate. If it does not, you just learned that a simple bandit is a worthy opponent, and your radio can go back to hunting for free airtime.
Frequently Asked Questions
Can reinforcement learning improve spectrum access over traditional methods?
In this simulator it can: a DQN with short observation history edges out an epsilon-greedy bandit when channels exhibit temporal correlation, since a channel that was just idle tends to stay idle. The gain is modest and depends on the transition probabilities drawn per episode.
Why is spectrum access a partially observable problem?
The secondary user only observes the channel it chooses to sense and transmit on. It never sees the full occupancy map of all channels simultaneously. We feed the agent a history of its own past observations to compensate for this partial observability.
What reward function works for dynamic spectrum access?
A simple +1 for successful transmission and a collision penalty (e.g., -1.0) is a clean starting point. The penalty weight controls how aggressively the agent protects primary users. Higher penalties make the agent more conservative, which matters for regulatory compliance.
Does the DQN need a recurrent policy for this problem?
Not necessarily. A fixed history window of 4–16 slots often suffices for the Markov channels in this simulator. For more complex PU traffic with long-range dependencies, switching to an LSTM policy from sb3-contrib can help, but it trains slower and is harder to debug.
How does this transfer to real radio hardware?
Not directly. Real systems add sensing errors, hardware switching latency, multi-user contention, and regulatory interference masks. Start with ns-3 or GNU Radio for higher-fidelity simulation, and treat the Markov toy as a formulation lesson, not a deployment model.
Sources
[1]Alexander M. Wyglinski, Maziar Nekovee and Y. Thomas Hu, Cognitive Radio Communications and Networks — spectrum sensing, dynamic access and regulatory foundations: Amazon search[2]Q. Zhao and B. M. Sadler, A Survey of Dynamic Spectrum Access — signal processing, networking and regulatory perspectives: ieeexplore.ieee.org/document/4153453[3]Y. Liu et al., Deep Reinforcement Learning for Dynamic Spectrum Access — DQN and actor-critic approaches for multi-channel access: arxiv.org/abs/1812.07538[4]ns-3 documentation — the discrete-event network simulator with LTE, Wi-Fi and spectrum modules: nsnam.org/documentation[5]GNU Radio — the open-source SDR toolkit for real-time spectrum sensing and waveform prototyping: gnuradio.org
