Stop Abusing Git for ML Data: The Architect’s Guide to AI Storage
Once you’ve done that, you’re ready to train a price model based on gradient-boosted trees (or maybe you’re going to fine-tune or train an LLM!). But you need to get the data in a format that’s ready to be used for training. You estimate you’ve got 500GB of training data. What do you do next? You’re going to train this in the cloud because you don’t have an H200 laying around. You’ve decided that a modern data storage format like Parquet is the way to go.
Throw your data into Git?
Like all other developers, you use Git to version control your code. This dataset is going to go through some iterations as you massage the data and clean it up, and other developers on your team will also be working on the data cleanup and preparation. Makes sense to also upload this dataset to Git, right? You’ve uploaded large files to Git and know about Git LFS (Large File Storage).
So you do exactly that, put this massive 500GB dataset into Git. Now your repo is HUGE, but Git-LFS just keeps a pointer to those large dataset files so you think everything is OK. But when you clone the repo in a new directory it takes 20 minutes. Your team’s/project’s slack channel is filled with messages about how the project’s CI/CD pipeline is taking f.o.r.e.v.e.r. , it’s basically ground to a halt.
Git works great for code and configuration files, has the ability to create a branching strategy that fits your team’s workflow, and is normally very fast. It is not so great at what you’re doing now: working with a large, mutable, in-progress dataset. Once your dataset is set in stone, it might be OK to store it in Git, but at this point in time it is just not going to work.
You’re going to be working on training this model, and likely other projects that involve a large dataset, quite a bit moving forward. You are contemplating what your “Raw Data to Trained Model Pipeline” should look like. Maybe your company has already built a data Lakehouse for various projects. Or maybe you’re like alot of organizations are just duct-taping storage for this kind of data. Let’s discuss what our current storage habits are breaking, and how we can fix them.
The Three Paradigms of ML Storage (and where they fail)
When building AI data pipelines, developers typically utilize one of three tools. Each has a major drawback when applied to modern machine learning pipelines.
Standard File Storage (POSIX)
This is just a standard file system found on your laptop or server. These file systems are backed by local NVMe/SSD drives, or storage arrays using spinning hard drives. You use the familiar OS commands like ls, cp, mv, etc. This kind of storage is super fast with fast disk-to-memory transfer speed and latency, and friction-free. It’s optimal for development tasks and responsive every-day usage. It’s also the best for using as a scratch disk during active GPU training.
This storage is great for compute and typical local development workflows, but it’s terrible for collaboration, scale, and modern (big) data catalogs. It’s also the most expensive kind of storage too. Moving data from this storage medium to the cloud is also slow and expensive.
Traditional Object Storage (AWS S3)
Most developers are familiar with S3 type of storage. It has massive capacity/scale and a flat object architecture. Access is typically a REST driven API or some kind of adapter. It’s also the current de facto standard for Big Data. If you’re using Iceberg, it’s likely the underlying foundation storage mechanism.
It does, however, have drawbacks. S3 is a silo - it’s not integrated at all into where your code lives. It’s behind a different security mechanism (IAM) so that makes integrating it into a ML process harder. You’ll also need to setup VPCs and pay egress fees to where ever you are doing the actual training. Updates typically mean you have to update an entire dataset file (files themselves are immutable, though an S3 bucket itself is, of course are). In a nutshell, it is not seamless.
Git-based Repositories
Let’s revisit Git as a place to store file for ML training. The good thing about Git is that we can nominally track our data files the same way we track our code, and everything can be in the same repo if needed. Hence, we have tight integration.
However, Git tracks changes by computing trees and hashes, but it doesn’t do it for the actual large file stored in Git-LFS, just the small pointer file. It does calculate and SHA-256 hash for the large file so it can track if it’s changed, which is stored in the pointer file. When you do a git diff on the repo, it just tells you that it’s been changed with no details, as it’s a large binary file, not text. Any change, small or large, means the entire file is replaced. And the previous version(s) are saved forever in your git repo (unless you manually prune it from the repo). There are ways to use Git-LFS to make it work better: GIT_LFS_SKIP_SMUDGE=1, lfs.fetchexclude, partial clone, and shallow clone… but now every new engineer needs three environment variables and to understand the nuance of these settings and features.
The Missing Link: Hugging Face Buckets
We want the scale and architecture of S3, we need for data stored for ML pipelines to be easily and efficiently (and quickly) updateable as we iterate on the data prep and other steps in our training process. Instead of forcing intermediate (in-flight, changing) datasets into a Git repo, or stuffing them into a silo’d S3 bucket, we can use Hugging Face Buckets. HF buckets fill the gap that ML training pipelines need to be fast and efficient, let’s see how.
Drop-in S3 API
There’s no new SDK to learn to use Hugging Face Buckets. They expose an S3 compatible endpoint, so all your existing tools will work out of the box. You point your clients (or code) to https://s3.hf.co , use your Hugging Face token (to create standard AWS-style access keys), and you’re off and running. Your AWS CLI, Apache Spark connector, etc all will work with these two changes.
The Secret Weapon: Xet Deduplication
Hugging Face Buckets was architected specifically for ML workflows, and at the heart of that is a Xet content-addressable storage backend. ML workflows generate highly repetitive artifacts (like checkpoints that are 90% identical to previous epochs), which Xet is able to deduce at the block level automatically. When working with your data before you start training, as you clean and organize things, Xet is able to minimize the amount of data that needs to be transferred to and from local working copies and ML training runs as needed.
The best part is that you don’t need to write any code to make this possible! Using the Hugging Face CLI or Python library in your code, it’s as simple as:
hf buckets sync . hf://buckets/prpatel/mybucket
And the mechanism of Xet will only download/upload the roughly 64KB average chunks which have changed. Think of that for a second: if you’ve only added a few rows into a big 1GB Parquet file, it will only send up that small change rather than uploading the entire 1GB. Hugging Face’s Parquet Content-Defined Chunking (CDC )implementation takes care of this for you automatically (you have to set a flag on the Parquet writer in Pandas/PyArrow). All while retaining S3 compatibility, so you have the flexibility to read this data from Apache Spark, Apache Arrow, or anything that supports the ubiquitous S3 API.
Versioned vs Mutable
One important thing to note is that HF Buckets are non-versioned, but are mutable. It’s designed for cases where you need simple, fast storage such as training checkpoints, logs, intermediate artifacts, or large file collections that don't need version control. However, you can promote the frozen dataset to a versioned Dataset repo when you’re done finalizing the data!
The Golden Rules of AI Storage
As you build out your next AI application or are doing data engineering, especially, but not limited to, AI/ML projects, keep these things in mind:
- Keep using Git where it makes sense: source code, configuration, etc
- Standard File storage for your usual development needs and as a scratch disk during GPU training runs
- Use Hugging Face Buckets for your data lakehouse, intermediate training checkpoints, agent traces, Parquet files, and everything else. Iterate your data set in the HF Bucket, then publish the final version in Git if you wish.
We haven’t discussed other things about Buckets such as CDN prewarming, low egress fees, as well as security & compliance, but you can read more on Buckets here and see the full guide. If you want a step-by-step tutorial and see how Buckets really is efficient (and fast!), I’ll be publishing a mini-workshop you can try out yourself soon!