Sam Austinon November 7, 202618 min read

Best Git Hosting and Large-File Storage Options for ML Projects

Best Git Hosting and Large-File Storage Options for ML Projects
Contents

Figure 1: Best Git Hosting and Large-File Storage — you push your repo, git push hangs forever because data/train.parquet is 4.2 GB. Git wasn't built for this. Neither was GitHub's 100 MB file limit.

You push your repo. git push origin main hangs. Forever. Turns out data/train.parquet is 4.2 GB. Git was not built for this. Neither was GitHub's 100 MB file limit.

Welcome to ML's dirty secret: code is small, data is huge.

I've migrated three projects off GitHub because of LFS bills. I've lost commits to GitLab's LFS bugs. I've watched DVC remotes silently corrupt. Let me save you the pain.

Here's the honest breakdown of every git hosting + large-file combo worth considering in 2024.

The Core Problem: Git ≠ Data

Git stores every version of every file. Forever. A 1 GB model file changed 50 times = 50 GB repo. Clone takes forever. CI chokes. Your teammates hate you.

Solutions exist. None are perfect. Pick your poison.


Tier 1: Git Hosting Platforms (The Code Layer)

GitHub (The Default)

Everyone's here. For good reason.

Plan Private Repos Storage (Git) LFS Included LFS Overage
Free Unlimited 1 GB/repo 1 GB $5/50 GB/mo
Pro Unlimited 2 GB/repo 2 GB $5/50 GB/mo
Team Unlimited 5 GB/repo 5 GB $5/50 GB/mo
Enterprise Unlimited 50 GB/repo 50 GB Custom

Pros: Best ecosystem. Actions, Codespaces, Dependabot, CodeQL, Projects, Discussions. Every tool integrates. Largest contributor pool.

Cons: LFS pricing is hostile. $5/50 GB/mo adds up fast. 100 MB hard limit on git files (not LFS). No self-hosted free tier.

LFS reality check: git lfs ls-files → sum sizes. Multiply by versions retained. That's your bill.

My take: Use for code. Keep data elsewhere. Or pay the tax.

GitLab (The Self-Hosted King)

If you might self-host, start here.

Plan Private Repos Storage (Git) LFS Included LFS Overage
Free Unlimited 5 GB/repo 5 GB $10/10 GB/mo
Premium Unlimited 10 GB/repo 10 GB $10/10 GB/mo
Ultimate Unlimited 25 GB/repo 25 GB $10/10 GB/mo
Self-Managed Unlimited Your disks Your disks $0

Pros: Self-managed = unlimited everything on your hardware. CI minutes generous (400k/mo free). Built-in registry, registry, pages. Better merge request UX.

Cons: LFS overage more expensive ($10/10 GB vs $5/50 GB). Cloud LFS has bugs (silent corruption reports). Smaller ecosystem. UI feels heavier.

Self-hosted reality: Omnibus package = 15 min install. Backups = your problem. Upgrades = your problem. But: zero limits, zero bills.

My take: Best if you have infra team or homelab. Cloud GitLab LFS? Avoid.

Bitbucket (The Atlassian Trap)

Only if you're already in Jira/Confluence hell.

Plan Private Repos Storage (Git) LFS Included LFS Overage
Free 5 users 1 GB/repo 1 GB Not sold separately
Standard Unlimited 5 GB/repo 5 GB Included in per-user
Premium Unlimited 10 GB/repo 10 GB Included in per-user

Pros: Jira integration is genuinely good. Built-in CI (Pipelines). Per-user pricing = predictable.

Cons: 5-user free limit. LFS bundled = pay for all users if one needs it. Slower UI. Fewer integrations.

Verdict: Only for Atlassian shops.

Sourcehut / Codeberg / Gitea (The Indie Options)

Minimalist. Ethical. Not for ML data.

Platform LFS Support Best For
Sourcehut No (philosophical) Pure code, mailing-list workflow
Codeberg Yes (Woodpecker CI) Open source, non-profit
Gitea (self-hosted) Yes Lightweight self-hosted

Use for: code-only side projects. Skip for: ML.


Tier 2: Large File Storage Solutions (The Data Layer)

This is where your models, datasets, checkpoints live.

DVC (Data Version Control) — The Standard

Git for data. Version control your data alongside code.

# Initialize
dvc init
dvc remote add -d myremote s3://my-bucket/dvc-store
# Or: gs://, az://, ssh://, /local/path, /nfs/path

# Track large files
dvc add data/train.parquet
git add data/train.parquet.dvc .gitignore
git commit -m "Add training data"

# Push data to remote
dvc push

# Clone + pull data (later)
git clone repo.git
cd repo
dvc pull

Storage backends: S3, GCS, Azure Blob, SSH, HDFS, WebDAV, local, NFS, OSS, Google Drive.

Pros: Native git integration. .dvc files are small text pointers. Pipeline reproduction (dvc repro). Experiments (dvc exp). Works with any git host. Open source.

Cons: Learning curve. Remote config per machine (or dvc config --global). No web UI (use DVC Studio or VS Code ext). Large file pulls = slow (single-threaded by default).

Pricing: Free (self-managed remote). DVC Studio: free for public, $15/mo private.

My take: The default choice for ML teams. I use it on every project.


Git LFS (Large File Storage) — The Native Option

GitHub/GitLab/Bitbucket built-in. Pointer files in git, blobs in LFS store.

# Track patterns
git lfs track "*.parquet"
git lfs track "models/*.pt"
git lfs track "data/**/*.npz"

# Add .gitattributes (auto-created)
git add .gitattributes
git commit -m "Configure LFS"

# Push normally
git push origin main

Pros: Zero config if host supports it. Works with existing git workflow. No extra tools.

Cons: Host-dependent pricing (see above). No pipeline/experiment features. Single remote only. Migration between hosts = painful. Partial clones not supported.

Pricing: See git hosting table above. GitHub: $5/50 GB/mo. GitLab: $10/10 GB/mo.

Verdict: Fine for <50 GB total. Expensive at scale. No DVC features.


lakeFS — Git-Like Object Store

S3 API + Git semantics. Branches, commits, merges on your data lake.

# Install
brew install lakefs/lakefs/lakectl

# Create repo
lakectl repo create lakefs://my-ml-data --storage-namespace s3://my-bucket/

# Commit data
lakectl commit lakefs://my-ml-data/main -m "Add v1 dataset"

# Branch for experiment
lakectl branch create lakefs://my-ml-data/experiment-1 lakefs://my-ml-data/main

# Merge back
lakectl merge lakefs://my-ml-data/experiment-1 lakefs://my-ml-data/main

Pros: True Git semantics on object storage. Zero-copy branches (metadata only). S3-compatible API (works with Spark, Pandas, Polars). Pre-signed URLs for direct access. Self-hosted or cloud.

Cons: Separate from code repo. Learning curve (new mental model). Cloud pricing: $0.10/GB/mo + requests. Self-hosted: run PostgreSQL + lakeFS.

Verdict: Excellent for data-lake-scale (TB+). Overkill for <100 GB.


Delta Lake / Iceberg / Hudi — Table Formats

Not version control. But versioned tables. If your data is tabular.

# Delta Lake example
import pandas as pd
from delta import DeltaTable

df = pd.read_parquet("s3://bucket/raw/data.parquet")
df.write.format("delta").mode("overwrite").save("s3://bucket/delta/table")

# Time travel
df_v1 = spark.read.format("delta").option("versionAsOf", 0).load("s3://bucket/delta/table")
df_latest = spark.read.format("delta").load("s3://bucket/delta/table")

Pros: ACID transactions. Time travel built-in. Schema evolution. Works with Spark, Flink, Trino, Polars, DuckDB. Cloud-managed (Databricks, Tabular, AWS Athena).

Cons: Tabular data only. Not for models, images, checkpoints. Requires compute engine for full features.

Verdict: Use alongside DVC/LFS for tabular datasets.


Tier 3: Artifact Registries (The Model Layer)

Models are not data. They're artifacts. Different access patterns.

Hugging Face Hub — The Model Standard

Git-based model hosting. Native transformers integration. Free for public.

from huggingface_hub import HfApi, snapshot_download

# Upload
api = HfApi()
api.upload_file(
    path_or_fileobj="model/pytorch_model.bin",
    path_in_repo="pytorch_model.bin",
    repo_id="username/my-model",
    repo_type="model",
)

# Download (with cache)
model = AutoModel.from_pretrained("username/my-model")

# Or raw
snapshot_download(repo_id="username/my-model", local_dir="./model")

Free tier: Unlimited public repos. Private repos: unlimited. 10 GB/file, 50 GB/repo. LFS: included.

Pros: Native HF ecosystem. Model cards, inference widget, spaces integration. Versioned via git (commit history = model versions). safetensors support. Inference API (free tier).

Cons: Git-based = clone downloads all versions. 10 GB/file limit. Not for datasets (use HF Datasets). No pipeline tracking.

Verdict: Best for sharing/distributing models. Not for training checkpoints.


Weights & Biases Artifacts / MLflow / ClearML

Experiment tracking + artifact storage.

# W&B Artifacts
import wandb
artifact = wandb.Artifact("my-model", type="model")
artifact.add_file("model/best.pt")
wandb.log_artifact(artifact)

# Later: use in another run
artifact = wandb.use_artifact("username/project/my-model:v3")
artifact.download()

Pricing (W&B): Free: 100 GB artifacts. Team: $15/user/mo, 1 TB. Enterprise: custom.

Pros: Linked to experiments/runs. Lineage tracking (which run produced this model). Promotion workflows (staging → production). API for CI/CD.

Cons: Vendor lock-in. Pricing at scale. Not a general file store.

Verdict: Use if you already use the platform. Don't adopt just for storage.


Model Registry Options Comparison

Feature HF Hub W&B MLflow ClearML
Git-based Yes No Optional No
Free tier Generous 100 GB Self-hosted Self-hosted
Lineage Basic Excellent Good Excellent
Promotion Manual Built-in Manual Built-in
Inference API Yes No Via serving Via serving
Best for Sharing models Experiment-driven teams MLOps platforms AutoML pipelines

Tier 4: Cloud-Native Storage (The "Just Works" Layer)

AWS: S3 + CodeCommit / GitHub + LFS

Service Use Case Pricing
S3 Standard DVC remote, model store $0.023/GB/mo
S3 Intelligent-Tiering Auto-optimize $0.023→$0.0125/GB/mo
S3 Glacier Archive old checkpoints $0.004/GB/mo
EFS Shared POSIX filesystem $0.30/GB/mo
FSx for Lustre High-perf ML training $0.15/GB/mo

CodeCommit: $1/mo/active user + $0.06/GB/mo. Git-compatible. Tiny free tier (5 users, 50 GB). Not worth it vs GitHub.

DVC + S3 = bulletproof. Versioned bucket = accidental delete protection. Lifecycle rules = auto-archive old versions.


GCP: Cloud Storage + Cloud Source Repositories

Service Use Case Pricing
Cloud Storage Standard DVC remote $0.020/GB/mo
Cloud Storage Nearline <1 access/mo $0.010/GB/mo
Cloud Storage Coldline <1 access/quarter $0.004/GB/mo
Filestore Shared NFS $0.20/GB/mo
Cloud Source Repos Git hosting $1/GB/mo (after 50 GB free)

Vertex AI integration: Managed datasets, model registry, lineage. If you're all-in on Vertex, use its artifact store.


Azure: Blob Storage + Azure Repos / GitHub

Service Use Case Pricing
Blob Hot Active data $0.0184/GB/mo
Blob Cool Infrequent $0.01/GB/mo
Blob Cold/Archive Rare $0.0015-0.0036/GB/mo
Azure Files SMB/NFS share $0.06-0.15/GB/mo
Azure Repos Git hosting Free (generous limits)

ML integration: Azure ML datastores, model registry, pipeline artifacts.


The Architecture I Actually Use (2024)

After migrating 5 projects, this is my settled stack:

┌─────────────────────────────────────────────────────────────┐
│                        GITHUB                                │
│  • Code (small, <100 MB)                                    │
│  • CI/CD (Actions)                                          │
│  • Issues, PRs, Discussions                                 │
│  • Docs (GitHub Pages)                                      │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│                         DVC + S3                             │
│  • Data (datasets, features)                                │
│  • Checkpoints (every N epochs)                             │
│  • Pipeline reproduction (dvc repro)                        │
│  • Experiments (dvc exp)                                    │
│  • Remote: versioned S3 bucket + lifecycle rules            │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│                      HUGGING FACE HUB                        │
│  • Final models (pushed from CI on tag)                     │
│  • Model cards with metrics                                 │
│  • Inference widget for demos                               │
│  • safetensors format                                       │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│                      WEIGHTS & BIASES                        │
│  • Experiment tracking                                      │
│  • Artifact lineage (which run → which model)               │
│  • Promotion: staging → production                          │
│  • Dashboards for stakeholders                              │
└─────────────────────────────────────────────────────────────┘

Monthly cost (typical project):

  • GitHub: $0 (public) or $4 (private Pro)
  • S3: $5-20 (depends on data size)
  • HF Hub: $0
  • W&B: $0 (academic) or $15/user (team)

Total: ~$20-40/mo. Scales to TB without architecture changes.


Migration Guide: Moving Between Systems

Git LFS → DVC

# 1. List LFS files
git lfs ls-files | awk '{print $3}' > lfs-files.txt

# 2. Untrack from LFS
cat lfs-files.txt | xargs git lfs untrack
git add .gitattributes
git commit -m "Remove LFS tracking"

# 3. Add to DVC
cat lfs-files.txt | xargs dvc add
git add *.dvc .gitignore
git commit -m "Add files to DVC"

# 4. Configure DVC remote
dvc remote add -d myremote s3://bucket/dvc
dvc push

GitHub LFS → Self-Hosted GitLab + MinIO

# 1. Mirror repo
git clone --mirror [email protected]:user/repo.git
cd repo.git
git push --mirror [email protected]:user/repo.git

# 2. Migrate LFS objects
git lfs fetch --all
git lfs push --all [email protected]:user/repo.git

# 3. Point DVC to MinIO
dvc remote add -d myremote s3://minio/dvc \
  --endpoint-url https://minio.selfhosted \
  --access-key-id $KEY --secret-access-key $SECRET
dvc push

Test thoroughly. Verify file hashes match. dvc status -c compares cache to remote.


Decision Matrix: Pick Your Stack

Your Situation Code Host Data Store Model Registry Experiment Tracking
Solo, public, <10 GB GitHub GitHub LFS HF Hub W&B (free)
Solo, private, <50 GB GitHub DVC + S3 HF Hub W&B (free)
Team, 50-500 GB GitHub DVC + S3 HF Hub + W&B Artifacts W&B
Team, 500 GB - 10 TB GitHub/GitLab DVC + S3/GCS W&B / MLflow W&B / MLflow
Enterprise, TB+, compliance GitLab Self-Managed lakeFS / Delta Lake MLflow / Custom MLflow / Custom
Academic, zero budget GitHub DVC + (univ storage / free tier S3) HF Hub W&B Academic / ClearML
All-in on cloud (AWS/GCP/Azure) GitHub Cloud Storage + DVC Vertex/SageMaker/Azure ML Vertex/SageMaker/Azure ML

Common Pitfalls (That Cost Me Weeks)

Pitfall Symptom Fix
LFS on GitHub, forgot to git lfs push Clones get pointer files, not data git lfs push --all origin in CI
DVC remote not configured on new machine dvc pull fails silently dvc remote add --global or docs
S3 versioning off Accidental overwrite = data loss Enable versioning + lifecycle rules
No .dvc in .gitignore Cache files committed to git echo "/.dvc" >> .gitignore
HF Hub 10 GB limit hit Push fails on large model Split model (sharded safetensors)
W&B artifacts not linked to runs "Which run made this model?" wandb.log_artifact(artifact, aliases=["best"])
Different DVC cache dir per user "Works on my machine" dvc config cache.dir /shared/cache

The "Just Start" Recommendation

Don't overthink it. Start here:

# 1. Code → GitHub (free, best ecosystem)
# 2. Data → DVC + S3 (or GCS, or Azure Blob)
#    dvc init
#    dvc remote add -d storage s3://my-bucket/dvc
#    dvc add data/
#    git commit -am "Add data tracking"
#    dvc push
# 3. Models → Hugging Face Hub (free, standard)
# 4. Experiments → Weights & Biases (free academic, $15 team)

This handles 95% of ML projects. Cost: ~$10-30/mo. Migrate individual pieces when you hit limits.

The best storage architecture is the one you actually use. Ship the model. 🚀


FAQ (Questions I Get Asked)

Q: "Can I just use Google Drive / Dropbox / OneDrive as DVC remote?"
A: Yes. dvc remote add -d myremote /mnt/gdrive/project/dvc. But: no versioning, no concurrent access, sync conflicts. Fine for solo. Bad for teams.

Q: "What about Git Annex?"
A: Predecessor to DVC. More complex, less ML-focused. DVC won. Use DVC.

Q: "Should I store data in the repo as .gitignored files?"
A: No. Clone won't get them. No versioning. No reproduction. Use DVC.

Q: "How do I share a 200 GB dataset with collaborators?"
A: DVC + S3 (requester pays) or HF Datasets. HF Datasets: datasets.push_to_hub(), collaborators load_dataset(). Free, versioned, streaming.

Q: "My CI fails because dvc pull is slow."
A: Shallow clone + sparse checkout. Cache DVC cache in CI. Use dvc fetch not dvc pull if you only need specific files.


Final Thoughts

Git hosting is solved. Use GitHub. Move on.

Large file storage is not solved. Every option has sharp edges.

DVC + object storage (S3/GCS/Azure) is the closest to "correct" for ML. It version-controls data with code. It reproduces pipelines. It scales to PB. It's open source.

Add Hugging Face Hub for model distribution. Add W&B/MLflow for experiment lineage. You now have a complete stack.

Stop reading storage comparisons. Start versioning your data. 😅

Questions? Migration horror stories? Better setups? Drop them below. I've moved 50 TB across 4 systems. Happy to compare scars. 🚀


Sources

What are You Looking For?

esc