Contents
Figure 1: Best Git Hosting and Large-File Storage — you push your repo, git push hangs forever because data/train.parquet is 4.2 GB. Git wasn't built for this. Neither was GitHub's 100 MB file limit.
You push your repo. git push origin main hangs. Forever. Turns out data/train.parquet is 4.2 GB. Git was not built for this. Neither was GitHub's 100 MB file limit.
Welcome to ML's dirty secret: code is small, data is huge.
I've migrated three projects off GitHub because of LFS bills. I've lost commits to GitLab's LFS bugs. I've watched DVC remotes silently corrupt. Let me save you the pain.
Here's the honest breakdown of every git hosting + large-file combo worth considering in 2024.
The Core Problem: Git ≠ Data
Git stores every version of every file. Forever. A 1 GB model file changed 50 times = 50 GB repo. Clone takes forever. CI chokes. Your teammates hate you.
Solutions exist. None are perfect. Pick your poison.
Tier 1: Git Hosting Platforms (The Code Layer)
GitHub (The Default)
Everyone's here. For good reason.
| Plan | Private Repos | Storage (Git) | LFS Included | LFS Overage |
|---|---|---|---|---|
| Free | Unlimited | 1 GB/repo | 1 GB | $5/50 GB/mo |
| Pro | Unlimited | 2 GB/repo | 2 GB | $5/50 GB/mo |
| Team | Unlimited | 5 GB/repo | 5 GB | $5/50 GB/mo |
| Enterprise | Unlimited | 50 GB/repo | 50 GB | Custom |
Pros: Best ecosystem. Actions, Codespaces, Dependabot, CodeQL, Projects, Discussions. Every tool integrates. Largest contributor pool.
Cons: LFS pricing is hostile. $5/50 GB/mo adds up fast. 100 MB hard limit on git files (not LFS). No self-hosted free tier.
LFS reality check: git lfs ls-files → sum sizes. Multiply by versions retained. That's your bill.
My take: Use for code. Keep data elsewhere. Or pay the tax.
GitLab (The Self-Hosted King)
If you might self-host, start here.
| Plan | Private Repos | Storage (Git) | LFS Included | LFS Overage |
|---|---|---|---|---|
| Free | Unlimited | 5 GB/repo | 5 GB | $10/10 GB/mo |
| Premium | Unlimited | 10 GB/repo | 10 GB | $10/10 GB/mo |
| Ultimate | Unlimited | 25 GB/repo | 25 GB | $10/10 GB/mo |
| Self-Managed | Unlimited | Your disks | Your disks | $0 |
Pros: Self-managed = unlimited everything on your hardware. CI minutes generous (400k/mo free). Built-in registry, registry, pages. Better merge request UX.
Cons: LFS overage more expensive ($10/10 GB vs $5/50 GB). Cloud LFS has bugs (silent corruption reports). Smaller ecosystem. UI feels heavier.
Self-hosted reality: Omnibus package = 15 min install. Backups = your problem. Upgrades = your problem. But: zero limits, zero bills.
My take: Best if you have infra team or homelab. Cloud GitLab LFS? Avoid.
Bitbucket (The Atlassian Trap)
Only if you're already in Jira/Confluence hell.
| Plan | Private Repos | Storage (Git) | LFS Included | LFS Overage |
|---|---|---|---|---|
| Free | 5 users | 1 GB/repo | 1 GB | Not sold separately |
| Standard | Unlimited | 5 GB/repo | 5 GB | Included in per-user |
| Premium | Unlimited | 10 GB/repo | 10 GB | Included in per-user |
Pros: Jira integration is genuinely good. Built-in CI (Pipelines). Per-user pricing = predictable.
Cons: 5-user free limit. LFS bundled = pay for all users if one needs it. Slower UI. Fewer integrations.
Verdict: Only for Atlassian shops.
Sourcehut / Codeberg / Gitea (The Indie Options)
Minimalist. Ethical. Not for ML data.
| Platform | LFS Support | Best For |
|---|---|---|
| Sourcehut | No (philosophical) | Pure code, mailing-list workflow |
| Codeberg | Yes (Woodpecker CI) | Open source, non-profit |
| Gitea (self-hosted) | Yes | Lightweight self-hosted |
Use for: code-only side projects. Skip for: ML.
Tier 2: Large File Storage Solutions (The Data Layer)
This is where your models, datasets, checkpoints live.
DVC (Data Version Control) — The Standard
Git for data. Version control your data alongside code.
# Initialize
dvc init
dvc remote add -d myremote s3://my-bucket/dvc-store
# Or: gs://, az://, ssh://, /local/path, /nfs/path
# Track large files
dvc add data/train.parquet
git add data/train.parquet.dvc .gitignore
git commit -m "Add training data"
# Push data to remote
dvc push
# Clone + pull data (later)
git clone repo.git
cd repo
dvc pull
Storage backends: S3, GCS, Azure Blob, SSH, HDFS, WebDAV, local, NFS, OSS, Google Drive.
Pros: Native git integration. .dvc files are small text pointers. Pipeline reproduction (dvc repro). Experiments (dvc exp). Works with any git host. Open source.
Cons: Learning curve. Remote config per machine (or dvc config --global). No web UI (use DVC Studio or VS Code ext). Large file pulls = slow (single-threaded by default).
Pricing: Free (self-managed remote). DVC Studio: free for public, $15/mo private.
My take: The default choice for ML teams. I use it on every project.
Git LFS (Large File Storage) — The Native Option
GitHub/GitLab/Bitbucket built-in. Pointer files in git, blobs in LFS store.
# Track patterns
git lfs track "*.parquet"
git lfs track "models/*.pt"
git lfs track "data/**/*.npz"
# Add .gitattributes (auto-created)
git add .gitattributes
git commit -m "Configure LFS"
# Push normally
git push origin main
Pros: Zero config if host supports it. Works with existing git workflow. No extra tools.
Cons: Host-dependent pricing (see above). No pipeline/experiment features. Single remote only. Migration between hosts = painful. Partial clones not supported.
Pricing: See git hosting table above. GitHub: $5/50 GB/mo. GitLab: $10/10 GB/mo.
Verdict: Fine for <50 GB total. Expensive at scale. No DVC features.
lakeFS — Git-Like Object Store
S3 API + Git semantics. Branches, commits, merges on your data lake.
# Install
brew install lakefs/lakefs/lakectl
# Create repo
lakectl repo create lakefs://my-ml-data --storage-namespace s3://my-bucket/
# Commit data
lakectl commit lakefs://my-ml-data/main -m "Add v1 dataset"
# Branch for experiment
lakectl branch create lakefs://my-ml-data/experiment-1 lakefs://my-ml-data/main
# Merge back
lakectl merge lakefs://my-ml-data/experiment-1 lakefs://my-ml-data/main
Pros: True Git semantics on object storage. Zero-copy branches (metadata only). S3-compatible API (works with Spark, Pandas, Polars). Pre-signed URLs for direct access. Self-hosted or cloud.
Cons: Separate from code repo. Learning curve (new mental model). Cloud pricing: $0.10/GB/mo + requests. Self-hosted: run PostgreSQL + lakeFS.
Verdict: Excellent for data-lake-scale (TB+). Overkill for <100 GB.
Delta Lake / Iceberg / Hudi — Table Formats
Not version control. But versioned tables. If your data is tabular.
# Delta Lake example
import pandas as pd
from delta import DeltaTable
df = pd.read_parquet("s3://bucket/raw/data.parquet")
df.write.format("delta").mode("overwrite").save("s3://bucket/delta/table")
# Time travel
df_v1 = spark.read.format("delta").option("versionAsOf", 0).load("s3://bucket/delta/table")
df_latest = spark.read.format("delta").load("s3://bucket/delta/table")
Pros: ACID transactions. Time travel built-in. Schema evolution. Works with Spark, Flink, Trino, Polars, DuckDB. Cloud-managed (Databricks, Tabular, AWS Athena).
Cons: Tabular data only. Not for models, images, checkpoints. Requires compute engine for full features.
Verdict: Use alongside DVC/LFS for tabular datasets.
Tier 3: Artifact Registries (The Model Layer)
Models are not data. They're artifacts. Different access patterns.
Hugging Face Hub — The Model Standard
Git-based model hosting. Native transformers integration. Free for public.
from huggingface_hub import HfApi, snapshot_download
# Upload
api = HfApi()
api.upload_file(
path_or_fileobj="model/pytorch_model.bin",
path_in_repo="pytorch_model.bin",
repo_id="username/my-model",
repo_type="model",
)
# Download (with cache)
model = AutoModel.from_pretrained("username/my-model")
# Or raw
snapshot_download(repo_id="username/my-model", local_dir="./model")
Free tier: Unlimited public repos. Private repos: unlimited. 10 GB/file, 50 GB/repo. LFS: included.
Pros: Native HF ecosystem. Model cards, inference widget, spaces integration. Versioned via git (commit history = model versions). safetensors support. Inference API (free tier).
Cons: Git-based = clone downloads all versions. 10 GB/file limit. Not for datasets (use HF Datasets). No pipeline tracking.
Verdict: Best for sharing/distributing models. Not for training checkpoints.
Weights & Biases Artifacts / MLflow / ClearML
Experiment tracking + artifact storage.
# W&B Artifacts
import wandb
artifact = wandb.Artifact("my-model", type="model")
artifact.add_file("model/best.pt")
wandb.log_artifact(artifact)
# Later: use in another run
artifact = wandb.use_artifact("username/project/my-model:v3")
artifact.download()
Pricing (W&B): Free: 100 GB artifacts. Team: $15/user/mo, 1 TB. Enterprise: custom.
Pros: Linked to experiments/runs. Lineage tracking (which run produced this model). Promotion workflows (staging → production). API for CI/CD.
Cons: Vendor lock-in. Pricing at scale. Not a general file store.
Verdict: Use if you already use the platform. Don't adopt just for storage.
Model Registry Options Comparison
| Feature | HF Hub | W&B | MLflow | ClearML |
|---|---|---|---|---|
| Git-based | Yes | No | Optional | No |
| Free tier | Generous | 100 GB | Self-hosted | Self-hosted |
| Lineage | Basic | Excellent | Good | Excellent |
| Promotion | Manual | Built-in | Manual | Built-in |
| Inference API | Yes | No | Via serving | Via serving |
| Best for | Sharing models | Experiment-driven teams | MLOps platforms | AutoML pipelines |
Tier 4: Cloud-Native Storage (The "Just Works" Layer)
AWS: S3 + CodeCommit / GitHub + LFS
| Service | Use Case | Pricing |
|---|---|---|
| S3 Standard | DVC remote, model store | $0.023/GB/mo |
| S3 Intelligent-Tiering | Auto-optimize | $0.023→$0.0125/GB/mo |
| S3 Glacier | Archive old checkpoints | $0.004/GB/mo |
| EFS | Shared POSIX filesystem | $0.30/GB/mo |
| FSx for Lustre | High-perf ML training | $0.15/GB/mo |
CodeCommit: $1/mo/active user + $0.06/GB/mo. Git-compatible. Tiny free tier (5 users, 50 GB). Not worth it vs GitHub.
DVC + S3 = bulletproof. Versioned bucket = accidental delete protection. Lifecycle rules = auto-archive old versions.
GCP: Cloud Storage + Cloud Source Repositories
| Service | Use Case | Pricing |
|---|---|---|
| Cloud Storage Standard | DVC remote | $0.020/GB/mo |
| Cloud Storage Nearline | <1 access/mo | $0.010/GB/mo |
| Cloud Storage Coldline | <1 access/quarter | $0.004/GB/mo |
| Filestore | Shared NFS | $0.20/GB/mo |
| Cloud Source Repos | Git hosting | $1/GB/mo (after 50 GB free) |
Vertex AI integration: Managed datasets, model registry, lineage. If you're all-in on Vertex, use its artifact store.
Azure: Blob Storage + Azure Repos / GitHub
| Service | Use Case | Pricing |
|---|---|---|
| Blob Hot | Active data | $0.0184/GB/mo |
| Blob Cool | Infrequent | $0.01/GB/mo |
| Blob Cold/Archive | Rare | $0.0015-0.0036/GB/mo |
| Azure Files | SMB/NFS share | $0.06-0.15/GB/mo |
| Azure Repos | Git hosting | Free (generous limits) |
ML integration: Azure ML datastores, model registry, pipeline artifacts.
The Architecture I Actually Use (2024)
After migrating 5 projects, this is my settled stack:
┌─────────────────────────────────────────────────────────────┐
│ GITHUB │
│ • Code (small, <100 MB) │
│ • CI/CD (Actions) │
│ • Issues, PRs, Discussions │
│ • Docs (GitHub Pages) │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ DVC + S3 │
│ • Data (datasets, features) │
│ • Checkpoints (every N epochs) │
│ • Pipeline reproduction (dvc repro) │
│ • Experiments (dvc exp) │
│ • Remote: versioned S3 bucket + lifecycle rules │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ HUGGING FACE HUB │
│ • Final models (pushed from CI on tag) │
│ • Model cards with metrics │
│ • Inference widget for demos │
│ • safetensors format │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ WEIGHTS & BIASES │
│ • Experiment tracking │
│ • Artifact lineage (which run → which model) │
│ • Promotion: staging → production │
│ • Dashboards for stakeholders │
└─────────────────────────────────────────────────────────────┘
Monthly cost (typical project):
- GitHub: $0 (public) or $4 (private Pro)
- S3: $5-20 (depends on data size)
- HF Hub: $0
- W&B: $0 (academic) or $15/user (team)
Total: ~$20-40/mo. Scales to TB without architecture changes.
Migration Guide: Moving Between Systems
Git LFS → DVC
# 1. List LFS files
git lfs ls-files | awk '{print $3}' > lfs-files.txt
# 2. Untrack from LFS
cat lfs-files.txt | xargs git lfs untrack
git add .gitattributes
git commit -m "Remove LFS tracking"
# 3. Add to DVC
cat lfs-files.txt | xargs dvc add
git add *.dvc .gitignore
git commit -m "Add files to DVC"
# 4. Configure DVC remote
dvc remote add -d myremote s3://bucket/dvc
dvc push
GitHub LFS → Self-Hosted GitLab + MinIO
# 1. Mirror repo
git clone --mirror [email protected]:user/repo.git
cd repo.git
git push --mirror [email protected]:user/repo.git
# 2. Migrate LFS objects
git lfs fetch --all
git lfs push --all [email protected]:user/repo.git
# 3. Point DVC to MinIO
dvc remote add -d myremote s3://minio/dvc \
--endpoint-url https://minio.selfhosted \
--access-key-id $KEY --secret-access-key $SECRET
dvc push
Test thoroughly. Verify file hashes match. dvc status -c compares cache to remote.
Decision Matrix: Pick Your Stack
| Your Situation | Code Host | Data Store | Model Registry | Experiment Tracking |
|---|---|---|---|---|
| Solo, public, <10 GB | GitHub | GitHub LFS | HF Hub | W&B (free) |
| Solo, private, <50 GB | GitHub | DVC + S3 | HF Hub | W&B (free) |
| Team, 50-500 GB | GitHub | DVC + S3 | HF Hub + W&B Artifacts | W&B |
| Team, 500 GB - 10 TB | GitHub/GitLab | DVC + S3/GCS | W&B / MLflow | W&B / MLflow |
| Enterprise, TB+, compliance | GitLab Self-Managed | lakeFS / Delta Lake | MLflow / Custom | MLflow / Custom |
| Academic, zero budget | GitHub | DVC + (univ storage / free tier S3) | HF Hub | W&B Academic / ClearML |
| All-in on cloud (AWS/GCP/Azure) | GitHub | Cloud Storage + DVC | Vertex/SageMaker/Azure ML | Vertex/SageMaker/Azure ML |
Common Pitfalls (That Cost Me Weeks)
| Pitfall | Symptom | Fix |
|---|---|---|
LFS on GitHub, forgot to git lfs push |
Clones get pointer files, not data | git lfs push --all origin in CI |
| DVC remote not configured on new machine | dvc pull fails silently |
dvc remote add --global or docs |
| S3 versioning off | Accidental overwrite = data loss | Enable versioning + lifecycle rules |
No .dvc in .gitignore |
Cache files committed to git | echo "/.dvc" >> .gitignore |
| HF Hub 10 GB limit hit | Push fails on large model | Split model (sharded safetensors) |
| W&B artifacts not linked to runs | "Which run made this model?" | wandb.log_artifact(artifact, aliases=["best"]) |
| Different DVC cache dir per user | "Works on my machine" | dvc config cache.dir /shared/cache |
The "Just Start" Recommendation
Don't overthink it. Start here:
# 1. Code → GitHub (free, best ecosystem)
# 2. Data → DVC + S3 (or GCS, or Azure Blob)
# dvc init
# dvc remote add -d storage s3://my-bucket/dvc
# dvc add data/
# git commit -am "Add data tracking"
# dvc push
# 3. Models → Hugging Face Hub (free, standard)
# 4. Experiments → Weights & Biases (free academic, $15 team)
This handles 95% of ML projects. Cost: ~$10-30/mo. Migrate individual pieces when you hit limits.
The best storage architecture is the one you actually use. Ship the model. 🚀
FAQ (Questions I Get Asked)
Q: "Can I just use Google Drive / Dropbox / OneDrive as DVC remote?"
A: Yes. dvc remote add -d myremote /mnt/gdrive/project/dvc. But: no versioning, no concurrent access, sync conflicts. Fine for solo. Bad for teams.
Q: "What about Git Annex?"
A: Predecessor to DVC. More complex, less ML-focused. DVC won. Use DVC.
Q: "Should I store data in the repo as .gitignored files?"
A: No. Clone won't get them. No versioning. No reproduction. Use DVC.
Q: "How do I share a 200 GB dataset with collaborators?"
A: DVC + S3 (requester pays) or HF Datasets. HF Datasets: datasets.push_to_hub(), collaborators load_dataset(). Free, versioned, streaming.
Q: "My CI fails because dvc pull is slow."
A: Shallow clone + sparse checkout. Cache DVC cache in CI. Use dvc fetch not dvc pull if you only need specific files.
Final Thoughts
Git hosting is solved. Use GitHub. Move on.
Large file storage is not solved. Every option has sharp edges.
DVC + object storage (S3/GCS/Azure) is the closest to "correct" for ML. It version-controls data with code. It reproduces pipelines. It scales to PB. It's open source.
Add Hugging Face Hub for model distribution. Add W&B/MLflow for experiment lineage. You now have a complete stack.
Stop reading storage comparisons. Start versioning your data. 😅
Questions? Migration horror stories? Better setups? Drop them below. I've moved 50 TB across 4 systems. Happy to compare scars. 🚀
Sources
[1]GitHub Documentation, Managing large files — LFS pricing, limits: docs.github.com[2]GitLab Documentation, Git LFS — LFS on GitLab: docs.gitlab.com[3]DVC Documentation, Get Started — DVC basics, remotes, pipelines: dvc.org/doc[4]lakeFS Documentation, Quickstart — Git-like object store: docs.lakefs.io[5]Hugging Face Hub, Model Repository — model hosting, versioning: huggingface.co/docs/hub[6]Weights & Biases, Artifacts Documentation — artifact versioning, lineage: docs.wandb.ai[7]MLflow Documentation, Model Registry — model versioning, staging: mlflow.org/docs[8]lakeFS Documentation, Overview — Git-like object store: docs.lakefs.io[9]AWS Documentation, S3 Pricing — storage classes, lifecycle: aws.amazon.com/s3/pricing[10]Google Cloud, Cloud Storage Pricing — storage classes, lifecycle: cloud.google.com/storage/pricing