WebOrganizer/FormatClassifier

1---2library_name: transformers3datasets:4- WebOrganizer/FormatAnnotations-Llama-3.1-8B5- WebOrganizer/FormatAnnotations-Llama-3.1-405B-FP86base_model:7- Alibaba-NLP/gte-base-en-v1.58---9# WebOrganizer/FormatClassifier10 11[[Paper](https://arxiv.org/abs/2502.10341)] [[Website](https://weborganizer.allenai.org)] [[GitHub](https://github.com/CodeCreator/WebOrganizer)]12 13The FormatClassifier organizes web content into 24 categories based on the URL and text contents of web pages.14The model is a [gte-base-en-v1.5](https://huggingface.co/Alibaba-NLP/gte-base-en-v1.5) with 140M parameters fine-tuned on the following training data:151. [WebOrganizer/FormatAnnotations-Llama-3.1-8B](https://huggingface.co/datasets/WebOrganizer/FormatAnnotations-Llama-3.1-8B): 1M documents annotated by Llama-3.1-8B (first-stage training)162. [WebOrganizer/FormatAnnotations-Llama-3.1-405B-FP8](https://huggingface.co/datasets/WebOrganizer/FormatAnnotations-Llama-3.1-405B-FP8): 100K documents annotated by Llama-3.1-405B-FP8 (second-stage training)17 18#### All Domain Classifiers19- [WebOrganizer/FormatClassifier](https://huggingface.co/WebOrganizer/FormatClassifier) *← you are here!*20- [WebOrganizer/FormatClassifier-NoURL](https://huggingface.co/WebOrganizer/FormatClassifier-NoURL)21- [WebOrganizer/TopicClassifier](https://huggingface.co/WebOrganizer/TopicClassifier)22- [WebOrganizer/TopicClassifier-NoURL](https://huggingface.co/WebOrganizer/TopicClassifier-NoURL)23 24## Usage25 26This classifier expects input in the following input format:27```28{url}29 30{text}31```32 33Example:34```python35from transformers import AutoTokenizer, AutoModelForSequenceClassification36 37tokenizer = AutoTokenizer.from_pretrained("WebOrganizer/FormatClassifier")38model = AutoModelForSequenceClassification.from_pretrained(39    "WebOrganizer/FormatClassifier",40    trust_remote_code=True,41    use_memory_efficient_attention=False)42 43web_page = """http://www.example.com44 45How to make a good sandwich? [Click here to read article]"""46 47inputs = tokenizer([web_page], return_tensors="pt")48outputs = model(**inputs)49 50probs = outputs.logits.softmax(dim=-1)51print(probs.argmax(dim=-1))52# -> 6 ("Truncated" format, which covers incomplete content)53```54 55You can convert the `logits` of the model with a softmax to obtain a probability distribution over the following 24 categories (in order of labels, also see `id2label` and `label2id` in the model config):561. Academic Writing572. Content Listing583. Creative Writing594. Customer Support605. Comment Section616. FAQ627. Truncated638. Knowledge Article649. Legal Notices6510. Listicle6611. News Article6712. Nonfiction Writing6813. About (Org.)6914. News (Org.)7015. About (Pers.)7116. Personal Blog7217. Product Page7318. Q&A Forum7419. Spam / Ads7520. Structured Data7621. Documentation7722. Audio Transcript7823. Tutorial7924. User Review80 81The full definitions of the categories can be found in the [taxonomy config](https://github.com/CodeCreator/WebOrganizer/blob/main/define_domains/taxonomies/formats.yaml).82 83#### Efficient Inference84We recommend that you use the efficient gte-base-en-v1.5 implementation by enabling unpadding and memory efficient attention. This __requires installing `xformers`__ (see more [here](https://huggingface.co/Alibaba-NLP/new-impl#recommendation-enable-unpadding-and-acceleration-with-xformers)) and loading the model like:85```python86AutoModelForSequenceClassification.from_pretrained(87    "WebOrganizer/FormatClassifier",88    trust_remote_code=True,89    unpad_inputs=True,90    use_memory_efficient_attention=True,91    torch_dtype=torch.bfloat1692)93```94 95 96## Citation97```bibtex98@article{wettig2025organize,99  title={Organize the Web: Constructing Domains Enhances Pre-Training Data Curation},100  author={Alexander Wettig and Kyle Lo and Sewon Min and Hannaneh Hajishirzi and Danqi Chen and Luca Soldaini},101  journal={arXiv preprint arXiv:2502.10341},102  year={2025}103}104```105