Contents
Figure 1: Best CI Services for ML Teams: Beyond GitHub Actions — the build that passes on a CPU runner and dies on a CUDA mismatch is the most expensive green check you will ever see
Your CI pipeline passes in four minutes on a CPU runner. Then the training job hits production, and a CUDA mismatch breaks it. Or your nightly regression run needs a GPU for two hours, and the hosted runner bill makes finance twitch. CI for ML teams differs from CI for web apps, because the expensive, flaky, slow part needs hardware that default runners do not provide.
I cannot claim I have run all of these in production, so this is an analysis from vendor documentation rather than war stories. Pricing and plans change constantly, so verify every number on the vendor's page before you commit. Picking the CI service matters less than deciding which jobs need a GPU at all.
What ML CI Actually Has to Do
Not every check needs the same hardware. Split the pipeline into tiers:
- Every commit, CPU, minutes: linting, type checks, unit tests, config validation.
- Every pull request, CPU or small GPU, minutes: data validation, smoke training on a tiny subset, shape checks.
- Nightly or on demand, GPU, hours: real training runs, benchmark evaluation, regression suites.
- Release, GPU plus deployment: model packaging, container builds, rollout gates.
Most teams burn money by running tier 3 work on every push. Ever watched a full training job trigger on a README typo? Tiering prevents that, and it is the single highest-leverage change you can make before comparing tools.
Why GitHub Actions Falls Short for Some ML Teams
GitHub Actions is fine for plenty of ML projects, and I would keep it if it fits. The walls are specific:
- GPU access. Standard hosted runners have no GPU. GitHub sells larger runners, and that tier includes GPU-powered machines [1], but they are paid and must be requested explicitly.
- Long jobs. Multi-hour training does not fit comfortably in a minutes-billed, ephemeral runner.
- Data locality. Pulling hundreds of gigabytes into a fresh runner every run wastes time and money.
- Cost control. Per-minute billing punishes slow, heavy jobs.
- Governance. Regulated teams may need CI inside their own network.
So the real question is not "which CI is best?" It is "where do my GPU jobs run, and who controls that capacity?"
Option 1: Keep Your CI, Add Self-Hosted GPU Runners
This changes the least. You keep the CI service for orchestration and register your own GPU machines as runners.
The machine needs NVIDIA drivers, the CUDA toolkit, and Docker with the NVIDIA Container Toolkit. Then you label it, and jobs target the label:
# GitHub Actions
jobs:
train:
runs-on: [self-hosted, gpu, linux]
steps:
- uses: actions/checkout@v4
- run: nvidia-smi
- run: python tests/smoke_train.py
# GitLab CI
gpu-job:
tags: [gpu]
script:
- nvidia-smi
- python tests/smoke_train.py
nvidia-smi in the first step is not ceremony. It is the cheapest way to catch a runner that registered correctly but cannot see its GPU.
CML for GPU runners and PR reports
The DVC team built CML (Continuous Machine Learning) for exactly this pattern: a thin wrapper over GitHub and GitLab runners that can spin up on-demand instances and post metrics and plots back to the pull request [5]. It is Apache-2.0, has about 4,200 stars, and was still being pushed in October 2026 [5], so the maintenance concern is smaller than it used to be.
CML is an add-on, not a CI service. You still need runners; CML makes the reporting and provisioning less painful.
The security warning
Self-hosted runners execute whatever your workflows tell them to. GitHub's hardening guide is blunt: self-hosted runners should almost never be used for public repositories, because anyone who can open a pull request can compromise the machine [2].
Never attach a persistent self-hosted runner to a public repo. Run jobs in containers or ephemeral VMs, and assume any runner that survives a job is carrying state you did not intend.
Option 2: GitLab CI
GitLab bundles pipelines, a container registry, and merge request integration in one platform. For ML teams, the useful parts are parent-child pipelines for decomposing a training workflow, and caching that cuts repeated dataset downloads.
GitLab now sells GPU-enabled hosted runners, which removes the "attach your own machine" step for smaller teams. The documented tag is saas-linux-medium-amd64-gpu-standard: 4 vCPUs, 15 GB RAM, 50 GB storage, and one NVIDIA Tesla T4 with 16 GB, available on GitLab.com for Premium and Ultimate customers [3].
That is a modest GPU, not a training cluster. It fits smoke tests, small fine-tunes, and evaluation jobs — which is exactly tier 2 and the low end of tier 3.
Self-managed GPU runners remain an option: configure the Docker executor to pass the GPU into containers, and you control the hardware.
Option 3: CircleCI
CircleCI aims at fast setup and minimal operations. Its GPU execution environment runs jobs on Linux or Windows VMs with NVIDIA GPUs [4]. On Linux you use the machine executor with a GPU-enabled image; the docs list GPU execution under the Scale plan [4].
That makes CircleCI a reasonable fit for a team that wants managed GPU capacity without owning the machines. Check the plan tier and per-minute pricing before you assume it is cheaper than a self-hosted runner — GPU minutes are never the cheap line on the invoice.
Option 4: Buildkite
Buildkite splits the system in two: it hosts the control plane, and you run agents on your own infrastructure. For ML, that is the attractive part — the agents live next to your data and GPUs, connect outbound over HTTPS, and keep source code and secrets inside your network.
You can keep agents warm between runs or make them ephemeral, and pipelines can generate steps at runtime, so a build can skip GPU work when only docs changed or fan evaluation out across datasets.
The cost is design work. Buildkite expects you to own scheduling, autoscaling, and image management. That is power for a platform team and a burden for a three-person ML group.
Option 5: Jenkins
Jenkins remains common in larger organizations because it runs anywhere and does anything. Shared libraries let many jobs reuse pipeline code, and manual job starts are one click in the GUI.
The trade-off is maintenance: plugins, upgrades, security patching, and a UI that has not aged well. Pick Jenkins when your organization already runs it. Starting fresh on Jenkins in 2026 is a choice you should be able to justify.
Option 6: Kubernetes-Native Pipelines
If your GPUs already live in Kubernetes, run CI there too. Each step maps to a pod or a container inside a pod, and the Kubernetes scheduler handles GPU allocation.
- Argo Workflows is a general workflow engine with native DAG support, used for CI/CD, data pipelines, and batch jobs. It pairs naturally with Argo CD for GitOps. It has no ML-specific abstractions, but it handles artifacts, GPU scheduling, and DAGs well.
- Tekton is a CI/CD framework built from Kubernetes primitives. Choose it when you are building a platform where teams bring their own pipelines; its UI is more basic than Argo's.
- Kubeflow Pipelines targets ML workflows specifically, and many teams find its complexity outweighs the benefit unless they are already invested.
- Airflow is a scheduler for data and training workflows, not a code-change validator. It complements CI rather than replacing it.
Do not adopt Kubernetes just for CI. The operational load will exceed the benefit unless you already run clusters for serving or batch work.
Where the GPU Capacity Comes From
Whichever CI you choose, the hardware question stays:
- On-premises machines: cheapest per hour if you keep them busy, with hardware and maintenance on you.
- Cloud on-demand GPUs: flexible and expensive.
- Spot or preemptible GPUs: cheaper, but jobs get interrupted, so design for retries and checkpoints.
- GPU rental providers: some pitch autoscaling ephemeral runners; treat the savings claims as marketing and benchmark your own workload. My GPU cloud comparison covers the options.
Use autoscaling ephemeral runners when you can. They start clean, shut down after the job, and keep idle cost near zero — which also removes the "forgotten GPU burning money overnight" failure mode.
Quick Comparison
- GitHub Actions plus self-hosted GPU runners: best if your team already lives on GitHub.
- GitLab CI: best integrated platform, with registry and merge request pipelines.
- CircleCI: best for managed CI with GPU executors and minimal operations.
- Buildkite: best for self-hosted GPU fleets and dynamic pipelines.
- Jenkins: best for organizations already invested in it.
- Argo Workflows: best Kubernetes-native option for ML DAGs plus CI.
- Tekton: best Kubernetes-native building blocks for platform teams.
- CML: best add-on for GPU runner provisioning and PR reports.
How to Choose
Start with four questions instead of a feature table.
Where do your GPUs live? Cloud GPUs and on-prem GPUs lead to different choices. Kubernetes points to Argo or Tekton; standalone machines point to Buildkite or self-hosted runners.
What does your Git host support? Moving CI away from your Git host adds integration work. Check webhook and status-check behaviour before you commit.
How much platform engineering can you afford? Managed CI saves time; self-hosted CI saves money and gives control. Match the tool to your real operations capacity.
What are your data and compliance rules? Regulated or client-owned data may need CI inside your network, which favours self-hosted agents or Kubernetes-native setups.
Design Tips That Matter More Than the Tool
- Cache aggressively. Dependency, model, and dataset caches cut minutes and dollars.
- Use small fixtures. Smoke tests should train on a tiny subset, not a sample of the real run.
- Pin environments. Containers with fixed CUDA, Python, and library versions prevent "works on my machine" bugs — the same reason I containerize research repos.
- Gate on metrics. Fail the build when accuracy drops or latency and memory spike past a threshold.
- Make results visible. Post metrics and plots to the pull request so reviewers see numbers, not just a green check.
- Tag GPU jobs. Route only the jobs that need GPUs to GPU runners.
- Set timeouts and budgets. A runaway job should not drain the account overnight.
- Keep data out of Git. Pull from object storage, not the repository.
Security Checklist for ML CI
- Run untrusted code in ephemeral containers or VMs.
- Never attach persistent self-hosted runners to public repos [2].
- Restrict secrets to protected branches and trusted workflows.
- Pin third-party actions and plugins to reviewed versions.
- Rotate credentials and prefer short-lived tokens.
- Limit network egress from runners.
- Audit who can edit pipeline files, because pipeline changes are code execution.
- Log runner activity and GPU usage.
The last two matter more than they look. A pipeline file is a program with your credentials attached, and GPU usage logs are how you find the job nobody remembers starting.
Common Pitfalls
- Running training on every push. Tier the pipeline.
- Trusting vendor comparisons. Most compare themselves favourably; run your own trial.
- Leaving GPU runners idle. Autoscale, or schedule them off.
- Ignoring CPU/GPU parity. A test that passes on CPU can fail on GPU — keep at least one smoke test on real hardware.
- Forgetting cost visibility. Track cost per pipeline and per job.
- Skipping flaky-test hygiene. Flaky GPU tests train teams to ignore red builds.
- Adopting Kubernetes only for CI. The operations load may exceed the benefit.
- No exit plan. Keep pipeline logic in portable scripts and containers, not platform-specific magic.
Most of these are the same mistake wearing different hats: running too much, too often, or too trustingly.
Best Book to Read
| Cover | Book | Description | Get it |
|---|---|---|---|
![]() |
The DevOps Handbook | the flow, feedback, and deployment-pipeline thinking that CI tools assume, written before GPUs made the bill interesting. | View on Amazon |
As an Amazon Associate I earn from qualifying purchases. Links on this page are affiliate links, and I may earn a commission at no extra cost to you.
Read it for the pipeline design, not the tooling. The chapters on small batches and fast feedback explain why tiering your ML jobs works.
Wrapping Up
There is no single best CI service for ML. Choose based on where your GPUs live, how much you want to operate, and how tightly you must control data. GitLab and CircleCI reduce operations work, Buildkite and Kubernetes-native tools reward platform investment, and self-hosted GPU runners bolt onto almost any CI.
Audit your pipeline this week: list every job, tag it by tier, and find the one that burns the most GPU time for the least value. Move it to a nightly schedule. If the bill drops and nobody notices, you just found free money. If someone notices, congratulations — you found a job that actually mattered.
Frequently Asked Questions
Does GitHub Actions support GPU runners?
Not on the standard hosted runners. GitHub sells larger runners, and that tier includes GPU-powered machines, but they are paid and must be selected explicitly. Most teams either use those or attach self-hosted GPU runners.
Is it safe to use a self-hosted runner on a public repository?
No. GitHub's own hardening guide says self-hosted runners should almost never be used for public repositories, because anyone who can open a pull request can run code on the machine. Use ephemeral containers or VMs, and keep persistent runners private.
Which CI service is best for ML?
There is no single winner. Pick by where your GPUs live: Kubernetes clusters point to Argo or Tekton, standalone GPU machines point to Buildkite or self-hosted runners, and teams that want managed GPU capacity should look at GitLab or CircleCI.
Do I need Kubernetes to run GPU CI?
No. Most teams start with one or two self-hosted GPU runners and a scheduled training job. Kubernetes pays off when you already run it for serving or batch work and want one scheduler for everything.
How do I stop CI costs from exploding?
Tier your jobs. Run lint, unit tests, and smoke training on CPU for every commit, and move full training and benchmark suites to a nightly schedule on GPU runners that shut down when idle.
Sources
[1]GitHub Docs, Larger runners — GPU-powered hosted runners and their capabilities: docs.github.com/en/actions/concepts/runners/larger-runners[2]GitHub Docs, Security hardening for GitHub Actions — why self-hosted runners should almost never serve public repositories: docs.github.com/en/actions/security-for-github-actions/security-guides/security-hardening-for-github-actions[3]GitLab Docs, GPU-enabled hosted runners — T4 machine type, tags, and plan tiers: docs.gitlab.com/ci/runners/hosted_runners/gpu_enabled[4]CircleCI Docs, Using the GPU execution environment — Linux and Windows GPU VMs and plan availability: circleci.com/docs/guides/execute/using-gpu[5]GitHub, iterative/cml — Continuous Machine Learning runner wrapper and PR reports, Apache-2.0: github.com/iterative/cml
