CoolFace
Datasetpublic

agentlans/en-document-format-classification

English Document Format Classification Dataset English-language web pages classified by document type, designed to train robust text classifiers and provide ready-to-use data for specific web formats. Purpose: Train generalized document classifiers or extract clean, single-format corpora for specific downstream tasks. Configurations: Each document type is available in its own dedicated dataset configuration (e.g., TutorialHow-ToGuide, PersonalAboutPage). Splits: The All… See the full description on the dataset page: https://huggingface.co/datasets/agentlans/en-document-format-classification.

sourceHugging Faceodc-byupdated 21d agoView on Hugging Face
0likes218downloads
Dataset Card

English Document Format Classification Dataset

English-language web pages classified by document type, designed to train robust text classifiers and provide ready-to-use data for specific web formats.

  • —Purpose: Train generalized document classifiers or extract clean, single-format corpora for specific downstream tasks.
  • —Configurations: Each document type is available in its own dedicated dataset configuration (e.g., TutorialHow-ToGuide, PersonalAboutPage).
  • —Splits: The All configuration contains every document type combined, featuring a 10% stratified test split.

Curation & Filtering Pipeline

This dataset is derived from a subset of the first 1 million rows of `allenai/c4`, utilizing annotations from `agentlans/en-document-classification`.

  1. 1.Filtering: Samples are included only when the weborganizer_format field matches doc_type_v2_primary.
  2. 2.Personally identifiable information (PII) removal: Samples were removed based on `agentlans/multilingual-e5-small-pii-detector`.

Class Distribution (All Split)

LabelTrainTestTotal
Academic Writing7,9138798,792
Content Listing10,2041,13411,338
Creative Writing2,9473283,275
Customer Support Page5,2375825,819
Discussion Forum / Comment Section3,8024234,225
FAQs1,5051671,672
Incomplete Content75884842
Knowledge Article17,1411,90519,046
Legal Notices3,3073683,675
Listicle4,5465055,051
News Article35,4013,93339,334
Nonfiction Writing83092922
Organizational About Page17,9531,99519,948
Organizational Announcement21,8762,43124,307
Personal About Page6,5147247,238
Personal Blog40,2594,47344,732
Product Page146,82716,314163,141
Q&A Forum4,0694524,521
Spam / Ads67774
Structured Data2,1102352,345
Technical Writing3,6434054,048
Transcript / Interview77786863
Tutorial / How-To Guide20,2442,24922,493
User Reviews6,5637297,292
Total364,49340,500404,993

Licensing

Distributed under the Open Data Commons Attribution License (ODC-BY), matching the licensing terms of upstream sources `allenai/c4` and `agentlans/en-document-classification`.