agentlans/en-document-format-classification

1---2language:3- en4license: odc-by5tags:6- document-classification7- text-classification8- web-pages9configs:10- config_name: AcademicWriting11  data_files:12  - path:13    - AcademicWriting.jsonl.zst14    split: train15- config_name: All16  default: true17  data_files:18  - path:19    - All/train.jsonl.zst20    split: train21  - path:22    - All/test.jsonl.zst23    split: test24- config_name: ContentListing25  data_files:26  - path:27    - ContentListing.jsonl.zst28    split: train29- config_name: CreativeWriting30  data_files:31  - path:32    - CreativeWriting.jsonl.zst33    split: train34- config_name: CustomerSupportPage35  data_files:36  - path:37    - CustomerSupportPage.jsonl.zst38    split: train39- config_name: DiscussionForumCommentSection40  data_files:41  - path:42    - DiscussionForumCommentSection.jsonl.zst43    split: train44- config_name: FAQs45  data_files:46  - path:47    - FAQs.jsonl.zst48    split: train49- config_name: IncompleteContent50  data_files:51  - path:52    - IncompleteContent.jsonl.zst53    split: train54- config_name: KnowledgeArticle55  data_files:56  - path:57    - KnowledgeArticle.jsonl.zst58    split: train59- config_name: LegalNotices60  data_files:61  - path:62    - LegalNotices.jsonl.zst63    split: train64- config_name: Listicle65  data_files:66  - path:67    - Listicle.jsonl.zst68    split: train69- config_name: NewsArticle70  data_files:71  - path:72    - NewsArticle.jsonl.zst73    split: train74- config_name: NonfictionWriting75  data_files:76  - path:77    - NonfictionWriting.jsonl.zst78    split: train79- config_name: OrganizationalAboutPage80  data_files:81  - path:82    - OrganizationalAboutPage.jsonl.zst83    split: train84- config_name: OrganizationalAnnouncement85  data_files:86  - path:87    - OrganizationalAnnouncement.jsonl.zst88    split: train89- config_name: PersonalAboutPage90  data_files:91  - path:92    - PersonalAboutPage.jsonl.zst93    split: train94- config_name: PersonalBlog95  data_files:96  - path:97    - PersonalBlog.jsonl.zst98    split: train99- config_name: ProductPage100  data_files:101  - path:102    - ProductPage.jsonl.zst103    split: train104- config_name: QAForum105  data_files:106  - path:107    - QAForum.jsonl.zst108    split: train109- config_name: SpamAds110  data_files:111  - path:112    - SpamAds.jsonl.zst113    split: train114- config_name: StructuredData115  data_files:116  - path:117    - StructuredData.jsonl.zst118    split: train119- config_name: TechnicalWriting120  data_files:121  - path:122    - TechnicalWriting.jsonl.zst123    split: train124- config_name: TranscriptInterview125  data_files:126  - path:127    - TranscriptInterview.jsonl.zst128    split: train129- config_name: TutorialHow-ToGuide130  data_files:131  - path:132    - TutorialHow-ToGuide.jsonl.zst133    split: train134- config_name: UserReviews135  data_files:136  - path:137    - UserReviews.jsonl.zst138    split: train139---140# English Document Format Classification Dataset141 142English-language web pages classified by document type, designed to train robust text classifiers and provide ready-to-use data for specific web formats.143 144* **Purpose:** Train generalized document classifiers or extract clean, single-format corpora for specific downstream tasks.145* **Configurations:** Each document type is available in its own dedicated dataset configuration (e.g., `TutorialHow-ToGuide`, `PersonalAboutPage`).146* **Splits:** The `All` configuration contains every document type combined, featuring a 10% stratified test split.147 148## Curation & Filtering Pipeline149 150This dataset is derived from a subset of the first 1 million rows of [`allenai/c4`](https://huggingface.co/datasets/allenai/c4), utilizing annotations from [`agentlans/en-document-classification`](https://huggingface.co/datasets/agentlans/en-document-classification).151 1521. **Filtering:** Samples are included only when the `weborganizer_format` field matches `doc_type_v2_primary`.1532. **Personally identifiable information (PII) removal:** Samples were removed based on [`agentlans/multilingual-e5-small-pii-detector`](https://huggingface.co/agentlans/multilingual-e5-small-pii-detector).154 155## Class Distribution (`All` Split)156 157| Label | Train | Test | Total |158| --- | --- | --- | --- |159| Academic Writing | 7,913 | 879 | 8,792 |160| Content Listing | 10,204 | 1,134 | 11,338 |161| Creative Writing | 2,947 | 328 | 3,275 |162| Customer Support Page | 5,237 | 582 | 5,819 |163| Discussion Forum / Comment Section | 3,802 | 423 | 4,225 |164| FAQs | 1,505 | 167 | 1,672 |165| Incomplete Content | 758 | 84 | 842 |166| Knowledge Article | 17,141 | 1,905 | 19,046 |167| Legal Notices | 3,307 | 368 | 3,675 |168| Listicle | 4,546 | 505 | 5,051 |169| News Article | 35,401 | 3,933 | 39,334 |170| Nonfiction Writing | 830 | 92 | 922 |171| Organizational About Page | 17,953 | 1,995 | 19,948 |172| Organizational Announcement | 21,876 | 2,431 | 24,307 |173| Personal About Page | 6,514 | 724 | 7,238 |174| Personal Blog | 40,259 | 4,473 | 44,732 |175| Product Page | 146,827 | 16,314 | 163,141 |176| Q&A Forum | 4,069 | 452 | 4,521 |177| Spam / Ads | 67 | 7 | 74 |178| Structured Data | 2,110 | 235 | 2,345 |179| Technical Writing | 3,643 | 405 | 4,048 |180| Transcript / Interview | 777 | 86 | 863 |181| Tutorial / How-To Guide | 20,244 | 2,249 | 22,493 |182| User Reviews | 6,563 | 729 | 7,292 |183| **Total** | **364,493** | **40,500** | **404,993** |184 185## Licensing186 187Distributed under the Open Data Commons Attribution License (ODC-BY), matching the licensing terms of upstream sources [`allenai/c4`](https://huggingface.co/datasets/allenai/c4) and [`agentlans/en-document-classification`](https://www.google.com/search?q=https://huggingface.co/datasets/agentlans/en-document-classification).188