agentlans/m-a-p-FineFineWeb-sample

Unofficial m-a-p/FineFineWeb Sample

This dataset is a processed, lightweight sample of the original m-a-p/FineFineWeb, a comprehensive corpus designed for fine-grained domain web text studies.

Dataset Description

• —Homepage: m-a-p/FineFineWeb
• —License: Apache 2.0

Sampling Methodology

To create this subset, the following processing steps were taken:

1. 1.Selection: 100 random .jsonl files were chosen from the original dataset.
2. 2.Extraction: 10,000 rows were downloaded per selected file.
3. 3.Processing: The extracted rows were combined and globally shuffled.
4. 4.Provenance: An additional file column was appended to each row to track the specific source file the data originated from.

Limitations

• —Task Specificity: The original dataset was designed primarily for content classification. It may not exhibit optimal quality or formatting for downstream tasks outside of this scope (e.g., general language modeling, text generation).
• —Temporal & Factuality Risks: As with any web-scraped data, the text reflects specific historical snapshots and may contain inaccurate, outdated, or biased information.

License & Citation

This sample is distributed under the Apache 2.0 license. If you use this sampled version in your research or applications, please cite the original authors:

@misc{map2024finefineweb,
  title={FineFineWeb: A Comprehensive Study on Fine-grained Domain Web Corpus},
  url={https://huggingface.co/datasets/m-a-p/FineFineWeb},
  author={Zhang, Ge and Du, Xinrun and Yu, Zhimiao and Wang, Zili and Wang, Zekun and Guo, Shuyue and Zheng, Tianyu and Zhu, Kang and Liu, Jerry and Yue, Shawn and Liu, Binbin and Peng, Zhongyuan and Yao, Yifan and Yang, Jack and Li, Ziming and Zhang, Bingni and Liu, Minghao and Liu, Tianyu and Gao, Yang and Chen, Wenhu and Zhou, Xiaohuan and Liu, Qian and Wang, Taifeng and Huang, Wenhao},
  publisher={Hugging Face},
  version={v0.1.0},
  month={December},
  year={2024}
}