Contents
Figure 1: Docker for Research Repos: Make Any ML GitHub Project Run the Same Everywhere — containers stacked like shipping freight, which is the whole idea: freeze the environment once and move it anywhere
Research repos break for boring reasons. The author pinned an old library, assumed a specific CUDA version, or forgot to list one system package — and your afternoon disappears into dependency hell. A container freezes the whole environment in one place, so the repo that worked on someone else's machine works on yours.
Docker does not make bad research code good. It makes good research code runnable. That distinction matters: before you containerize a repo, check whether it is worth running at all — the same rubric I use to judge a GitHub ML repo applies here.
Why Docker Fits Research Code
You get three wins, and they are all about time:
- Reproducibility: everyone runs the same OS, Python, and library versions, including the CUDA stack.
- Isolation: an old PyTorch build never fights the project you are paid for.
- Portability: you move an image instead of a pile of setup instructions.
Ever spent an afternoon matching a paper's exact PyTorch version? A Dockerfile ends that ritual. It also gives you something a requirements.txt never does: a record of the system libraries, which is where most "it works on my machine" failures actually live.
What You Need on the Host
Docker alone cannot touch your GPU. You also need the NVIDIA driver and the NVIDIA Container Toolkit on the host — it ships the runtime hook and utilities that configure containers to use NVIDIA GPUs, a job Docker does not do by default [1].
After installing the toolkit, register it with Docker:
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
Those two commands are the step most tutorials skip, and without them --gpus all silently gives you a CPU container. Follow NVIDIA's current install guide for your distribution, because package names differ between Ubuntu, RHEL, and WSL [1].
Pick a Base Image
Your base image decides your CUDA and Python versions. There are two common choices:
- A framework image, such as
pytorch/pytorchwith a CUDA tag. PyTorch is preinstalled, so you start fast [2]. - A CUDA image, such as
nvidia/cuda, and you install the framework yourself. More control, more work.
I usually start with the framework image. Match the tag to what the repo expects: read its README for the PyTorch and CUDA versions, then pick the closest image.
One rule matters more than the rest: the container's CUDA version must not exceed the host driver's supported version. If the image needs a newer driver than you have, use an older CUDA tag rather than upgrading a working machine [4].
A Dockerfile You Can Adapt
Here is the template I start from. Swap the base tag for one that matches your repo:
FROM pytorch/pytorch:<TAG-MATCHING-YOUR-REPO>
# System packages many repos quietly need
RUN apt-get update && apt-get install -y --no-install-recommends \
git ffmpeg libgl1 \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /workspace
# Copy requirements first so Docker caches this layer
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
# Copy the rest of the code
COPY . .
CMD ["python", "train.py"]
Copying requirements.txt before the code is the one trick that pays for itself. Docker reuses the cached install layer whenever only your code changes, which turns a five-minute rebuild into five seconds.
The apt-get line covers system libraries that requirements files almost never mention — OpenCV needs libgl1, video pipelines need ffmpeg. Add a .dockerignore with .git, datasets, and checkpoints, or your build context quietly ships gigabytes to the daemon on every build [5].
Build and Run
Build the image once:
docker build -t my-ml-repo .
Then run it with the GPU enabled:
docker run --rm -it --gpus all \
--shm-size=16g \
-v $(pwd)/data:/workspace/data \
-v $(pwd)/outputs:/workspace/outputs \
my-ml-repo
Omitting --gpus all is the mistake everyone makes once. The container then cannot see the host GPUs, even when the image contains every CUDA library. You can also limit access: --gpus '"device=0,1"' gives the container only those two GPUs.
--shm-size matters more than it looks. PyTorch data loaders use shared memory, and the default 64 MB is not enough for multi-worker loading — you get a crash with a confusing bus error instead of a clear out-of-memory message. Some PyTorch images recommend --ipc=host for the same reason [2]; I prefer an explicit --shm-size because it does not share the host's entire IPC namespace.
Then confirm the container actually sees the GPU:
import torch
print(torch.cuda.is_available())
That should print True. If it prints False, the problem is almost always the toolkit registration from the previous section, not your Dockerfile.
Handle Data and Checkpoints With Volumes
Never bake datasets or model weights into your image. The image gets huge, and every checkpoint dies with the container. Mount folders instead — I mount data/ and outputs/ separately so it is obvious what persists.
File ownership trips people up. Files written by a container usually belong to root on the host. Pass --user="$(id -u):$(id -g)" and you will never run sudo chown on your results again.
Docker Compose for Repeatable Runs
Long docker run commands get ugly and get retyped wrong. A Compose file records the flags once:
services:
trainer:
build: .
shm_size: "16gb"
volumes:
- ./data:/workspace/data
- ./outputs:/workspace/outputs
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
Then docker compose up reproduces the whole run. GPU reservations need Compose v2 plus the NVIDIA Container Toolkit [3]; if the GPU does not appear, check docker compose version before you edit the YAML.
This is the same pattern I use for the multi-container ML dev setup, just narrowed to one training service.
Make It Truly Reproducible
A Dockerfile alone does not guarantee identical results. Add these habits:
- Pin exact versions. Write
torch==2.4.1, nottorch. Better, commit a lock file. - Pin the base image tag. Avoid
latest, because it changes under you. - Record the image ID next to your experiment results, so a number can be traced back to a build.
- Set random seeds in the code. Docker fixes the environment, not the randomness.
- Keep the Dockerfile in the repo and version it with the code.
That sounds like extra work until the first time a result cannot be reproduced. Ten minutes now saves a week later.
Common Pitfalls
- GPU not visible in the container. Check
--gpus all, confirmnvidia-smiworks on the host, then confirm the toolkit is registered with Docker [1]. - CUDA version errors. Pick a base image whose CUDA version your driver supports; the container cannot outrun the host driver [4].
- Permission denied on
/dev/nvidia*. Check device permissions and whether your user is in thedockergroup. - Out-of-memory during data loading. Raise
--shm-sizebefore you blame the model. - Build fails on a missing system library. Add it to the
apt-get installline —libgl1andffmpegcover most of them. - Using
latesttags. Your build works today and breaks next month with no code change. - Copying data into the image. Mount a volume; images are for code, not datasets.
The meta-mistake is containerizing everything. Docker adds a build step, an image registry, and a layer of indirection. For a single script with three dependencies, a virtual environment is still the right answer. Once the project has CUDA, system libraries, and a collaborator, the container earns its keep — which is also the point where you should think about shipping the model, not just training it.
Best Book to Read
| Cover | Book | Description | Get it |
|---|---|---|---|
![]() |
Docker: Up & Running | the operations side the tutorials skip: image layers, registries, Compose, and how to keep builds small and predictable. | View on Amazon |
As an Amazon Associate I earn from qualifying purchases. Links on this page are affiliate links, and I may earn a commission at no extra cost to you.
Read it after your first working Dockerfile, not before. The commands will make sense once you have watched a layer cache do its job.
Wrapping Up
Docker makes research repos portable. Install the NVIDIA Container Toolkit, pick a base image that matches the repo, write a short Dockerfile, mount data as volumes, and pass --gpus all. Add Compose when the commands grow long.
Take one repo that gave you trouble and containerize it this week. The first Dockerfile takes an hour; the second takes ten minutes. Which stubborn repo will you tame first?
Frequently Asked Questions
Do I need an NVIDIA GPU to use Docker for ML repos?
No. Docker works fine for CPU-only ML projects, and gradient-boosted models often run faster on the host anyway. GPU support needs the NVIDIA driver plus the NVIDIA Container Toolkit on the host, but a CPU container runs anywhere Docker is installed.
Can I use Docker on a Mac for ML projects?
Yes for CPU containers. NVIDIA GPU passthrough does not exist on macOS, so CUDA workloads need a Linux host, a cloud GPU, or Colab. Apple Silicon can run CPU images, but check the image has an arm64 build or it will fall back to slow emulation.
How do I prevent permission issues with mounted volumes?
Pass --user="$(id -u):$(id -g)" when you run the container so files are written as your own user, or create a non-root user in the Dockerfile. Otherwise outputs land on the host owned by root and you end up running sudo chown on your results.
Should I use Docker Compose or plain docker run?
Use docker run while you are still changing flags. Move to Compose once the command is stable, because the GPU reservation, shared memory size, and volume mounts live in a file you can commit and reuse.
How do I keep a Docker build reproducible?
Pin the base image tag instead of latest, pin exact package versions or commit a lock file, record the image ID next to your experiment results, and set random seeds in the code. Docker fixes the environment, not the randomness.
Sources
[1]NVIDIA, NVIDIA Container Toolkit — install guide: toolkit installation and thenvidia-ctk runtime configure --runtime=dockerstep — docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html[2]Docker Hub, pytorch/pytorch image: CUDA and Python tags, plus the recommended run flags — hub.docker.com/r/pytorch/pytorch[3]Docker Docs, Compose GPU support: device reservations and thedeploy.resources.reservations.devicesschema — docs.docker.com/compose/how-tos/gpu-support[4]NVIDIA, CUDA compatibility: how container CUDA versions relate to the host driver — docs.nvidia.com/deploy/cuda-compatibility[5]Docker Docs, Build context and.dockerignore: what gets sent to the builder and how to exclude it — docs.docker.com/build/building/context
Related Articles
- Dockerize an LLM Application for Production
- Docker Compose for ML Projects: Multi-Container Development Setup
- LiteLLM Docker Production Deployment
- Best GitHub Repos for MLOps
