open-athena/snowball-replay
Snowball Hugging Face row replay This repository identifies the upstream Hugging Face rows retained in the Snowball pretraining store. It contains row locators and mixture metadata, not source documents or token arrays. A reader does not need Marin, private GCS access, or either of the earlier Snowball index repositories. Get the selected rows Install Python 3.12 or newer, then run: python -m pip install huggingface_hub pyarrow zstandard hf download… See the full description on the dataset page: https://huggingface.co/datasets/open-athena/snowball-replay.
Fix replay quick start
Clarify replay instructions
Upload data/source_id=00083/part-hash14-00000.parquet through validate.py
Upload data/source_id=00080/part-00001.parquet through data/source_id=00083/part-hash13-00000.parquet
Upload data/source_id=00077/part-hash14-00000.parquet through data/source_id=00080/part-00000.parquet
Upload data/source_id=00077/part-hash07-00000.parquet through data/source_id=00077/part-hash13-00002.parquet
Upload data/source_id=00077/part-hash00-00000.parquet through data/source_id=00077/part-hash06-00003.parquet
Upload data/source_id=00074/part-hash07-00000.parquet through data/source_id=00076/part-hash15-00000.parquet
Upload data/source_id=00073/part-hash15-00004.parquet through data/source_id=00074/part-hash06-00000.parquet
Upload data/source_id=00073/part-hash14-00006.parquet through data/source_id=00073/part-hash15-00003.parquet
Upload data/source_id=00073/part-hash13-00009.parquet through data/source_id=00073/part-hash14-00005.parquet
Upload data/source_id=00073/part-hash12-00013.parquet through data/source_id=00073/part-hash13-00008.parquet
Upload data/source_id=00073/part-hash11-00014.parquet through data/source_id=00073/part-hash12-00012.parquet
Upload data/source_id=00073/part-hash10-00016.parquet through data/source_id=00073/part-hash11-00013.parquet
Upload data/source_id=00073/part-hash10-00002.parquet through data/source_id=00073/part-hash10-00015.parquet
Upload data/source_id=00073/part-hash09-00005.parquet through data/source_id=00073/part-hash10-00001.parquet
Upload data/source_id=00073/part-hash08-00006.parquet through data/source_id=00073/part-hash09-00004.parquet
Upload data/source_id=00073/part-hash07-00009.parquet through data/source_id=00073/part-hash08-00005.parquet
Upload data/source_id=00073/part-hash06-00012.parquet through data/source_id=00073/part-hash07-00008.parquet
Upload data/source_id=00073/part-hash05-00015.parquet through data/source_id=00073/part-hash06-00011.parquet
Upload data/source_id=00073/part-hash04-00017.parquet through data/source_id=00073/part-hash05-00014.parquet
Upload data/source_id=00073/part-hash04-00002.parquet through data/source_id=00073/part-hash04-00016.parquet
Upload data/source_id=00073/part-hash03-00004.parquet through data/source_id=00073/part-hash04-00001.parquet
Upload data/source_id=00073/part-hash02-00006.parquet through data/source_id=00073/part-hash03-00003.parquet
Upload data/source_id=00073/part-hash01-00010.parquet through data/source_id=00073/part-hash02-00005.parquet
Upload data/source_id=00073/part-hash00-00012.parquet through data/source_id=00073/part-hash01-00009.parquet
Upload data/source_id=00071/part-hash14-00001.parquet through data/source_id=00073/part-hash00-00011.parquet
Upload data/source_id=00071/part-hash04-00000.parquet through data/source_id=00071/part-hash14-00000.parquet
Upload data/source_id=00069/part-hash07-00000.parquet through data/source_id=00071/part-hash03-00001.parquet
Upload data/source_id=00066/part-hash13-00001.parquet through data/source_id=00069/part-hash06-00000.parquet
Upload data/source_id=00066/part-hash08-00001.parquet through data/source_id=00066/part-hash13-00000.parquet
Upload data/source_id=00066/part-hash03-00002.parquet through data/source_id=00066/part-hash08-00000.parquet
Upload data/source_id=00065/part-hash15-00009.parquet through data/source_id=00066/part-hash03-00001.parquet
Upload data/source_id=00065/part-hash14-00005.parquet through data/source_id=00065/part-hash15-00008.parquet
Upload data/source_id=00065/part-hash13-00001.parquet through data/source_id=00065/part-hash14-00004.parquet
Upload data/source_id=00065/part-hash11-00010.parquet through data/source_id=00065/part-hash13-00000.parquet
Upload data/source_id=00065/part-hash10-00007.parquet through data/source_id=00065/part-hash11-00009.parquet
Upload data/source_id=00065/part-hash09-00004.parquet through data/source_id=00065/part-hash10-00006.parquet
Upload data/source_id=00065/part-hash08-00000.parquet through data/source_id=00065/part-hash09-00003.parquet
Upload data/source_id=00065/part-hash06-00009.parquet through data/source_id=00065/part-hash07-00011.parquet
Upload data/source_id=00065/part-hash05-00008.parquet through data/source_id=00065/part-hash06-00008.parquet
Upload data/source_id=00065/part-hash04-00002.parquet through data/source_id=00065/part-hash05-00007.parquet
Upload data/source_id=00065/part-hash03-00000.parquet through data/source_id=00065/part-hash04-00001.parquet
Upload data/source_id=00065/part-hash01-00010.parquet through data/source_id=00065/part-hash02-00008.parquet
Upload data/source_id=00065/part-hash00-00007.parquet through data/source_id=00065/part-hash01-00009.parquet
Upload data/source_id=00064/part-hash10-00001.parquet through data/source_id=00065/part-hash00-00006.parquet
Upload data/source_id=00064/part-hash01-00001.parquet through data/source_id=00064/part-hash10-00000.parquet
Upload data/source_id=00063/part-hash15-00008.parquet through data/source_id=00064/part-hash01-00000.parquet
Upload data/source_id=00063/part-hash14-00012.parquet through data/source_id=00063/part-hash15-00007.parquet
Upload data/source_id=00063/part-hash13-00017.parquet through data/source_id=00063/part-hash14-00011.parquet