Did you know DVC doesn’t handle large datasets? Neither did we and here’s how we fixed it

We hit a nice roadblock yesterday while working on a dataset with 5 million samples. It took us 8 hours to upload the dataset and 11 hours to download and unpack it. We learned some valuable lessons that Willem wrote down on his blog.

This one is worth a read if you’re working with larger datasets in a machine-learning environment!