CoolFace
Datasetpublic

agentlans/en-document-classification

English Document Classification Dataset This dataset provides a curated subset of the first 1 million rows from the allenai/c4 (English configuration), enriched with multi-perspective topic annotations. It is designed for researchers exploring document classification, domain adaptation, and label noise in massive web-crawled corpora. Dataset Summary The dataset integrates predictions from classification models to provide a holistic view of each document’s content.… See the full description on the dataset page: https://huggingface.co/datasets/agentlans/en-document-classification.

sourceHugging Faceodc-byupdated 22d agoView on Hugging Face
1likes426downloads
Dataset Card

English Document Classification Dataset

This dataset provides a curated subset of the first 1 million rows from the allenai/c4 (English configuration), enriched with multi-perspective topic annotations. It is designed for researchers exploring document classification, domain adaptation, and label noise in massive web-crawled corpora.

Dataset Summary

The dataset integrates predictions from classification models to provide a holistic view of each document’s content. To ensure utility, the data was stratified across these variables and split into 80/10/10% (Train/Validation/Test) sets, organized into specific configurations based on the source classifier.

Classifier Metadata & Schema

The following classifiers were used to generate the annotation columns:

ClassifierColumn NameDescription
`nvidia/domain-classifier`nvidia_domainGeneral web domain categorization.
`agentlans/GIST-small-weborganizer-topic`weborganizer_topicGeneral web topic categorization.
`agentlans/GIST-small-weborganizer-format`weborganizer_formatGeneral web format categorization.
`agentlans/GIST-small-finefineweb`finefinewebFine-grained domain web corpus.
`BAAI/IndustryCorpus2_Classifier`industry_corpus2General industry categorization.
`classla/multilingual-IPTC-news-topic-classifier`classla_newsStandardized news industry topic codes.
`classla/ParlaCAP-Topic-Classifier`classla_ParlaCAPLegislative and parliamentary topic categories.
`cardiffnlp/tweet-topic-latest-multi`cardiffnlp_tweetSocial-media style topic classification.
`ibm-granite/GneissWeb.*_classifier`gneissScience, technology, education, or medical classification.
`EssentialAI/eai-distill-0.5b`fdc_label and other columnsCategorical mapping derived from FDC data and content analysis.
[!NOTE] FDC (Free Decimal Correspondence) Mapping: Labels were derived from predictions by EssentialAI/eai-distill-0.5b. Call numbers were truncated to the nearest multiple of 10 and mapped to official FDC categories.

Quality Indicators

  • —`pristine` column: A boolean flag based on data generated by the EAI classifier. It indicates whether the document is structurally complete or likely missing content/context.
  • —Configurations: The dataset is partitioned into configs (e.g., classla_ParlaCAP) to allow users to load data stratified by specific model outputs.

Limitations & Biases

Beware of the following caveats in automated labelling:

  • —Class Imbalance: Significant distribution shifts exist between categories, reflecting the natural frequency of topics in the C4 corpus.
  • —Silver-Standard Labels: All labels are model-generated (silver-standard). Errors or biases present in the source classifiers will be reflected in this dataset.
  • —Label Ambiguity: Web documents often overlap multiple domains (for example, a personal blog discussing both geopolitics and cooking). Single-label assignment may oversimplify these documents.

Licensing

This dataset is released under the Open Data Commons Attribution License (ODC-BY). Please attribute the source models and the creators of the allenai/c4 dataset when using this resource.