datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
memmaped_2048len_PretrainGiftEvalmemmap-obelicsfineweb-edu-500m-memmapThe file is memory-mapped: data remains on disk and the OS loads only the accessed pages into the page cache when slices are requested. Python only holds views to this data through pointers, not full copies, keeping memory usage low and stable across iterations.
Memmap