WebOrganizer/TopicClassifier

1---2library_name: transformers3datasets:4- WebOrganizer/TopicAnnotations-Llama-3.1-8B5- WebOrganizer/TopicAnnotations-Llama-3.1-405B-FP86base_model:7- Alibaba-NLP/gte-base-en-v1.58---9# WebOrganizer/TopicClassifier10 11[[Paper](https://arxiv.org/abs/2502.10341)] [[Website](https://weborganizer.allenai.org)] [[GitHub](https://github.com/CodeCreator/WebOrganizer)]12 13The TopicClassifier organizes web content into 24 categories based on the URL and text contents of web pages.14The model is a [gte-base-en-v1.5](https://huggingface.co/Alibaba-NLP/gte-base-en-v1.5) with 140M parameters fine-tuned on the following training data:151. [WebOrganizer/TopicAnnotations-Llama-3.1-8B](https://huggingface.co/datasets/WebOrganizer/TopicAnnotations-Llama-3.1-8B): 1M documents annotated by Llama-3.1-8B (first-stage training)162. [WebOrganizer/TopicAnnotations-Llama-3.1-405B-FP8](https://huggingface.co/datasets/WebOrganizer/TopicAnnotations-Llama-3.1-405B-FP8): 100K documents annotated by Llama-3.1-405B-FP8 (second-stage training)17 18#### All Domain Classifiers19- [WebOrganizer/FormatClassifier](https://huggingface.co/WebOrganizer/FormatClassifier)20- [WebOrganizer/FormatClassifier-NoURL](https://huggingface.co/WebOrganizer/FormatClassifier-NoURL)21- [WebOrganizer/TopicClassifier](https://huggingface.co/WebOrganizer/TopicClassifier) *← you are here!*22- [WebOrganizer/TopicClassifier-NoURL](https://huggingface.co/WebOrganizer/TopicClassifier-NoURL)23 24## Usage25 26This classifier expects input in the following input format:27```28{url}29 30{text}31```32 33Example:34```python35from transformers import AutoTokenizer, AutoModelForSequenceClassification36 37tokenizer = AutoTokenizer.from_pretrained("WebOrganizer/TopicClassifier")38model = AutoModelForSequenceClassification.from_pretrained(39    "WebOrganizer/TopicClassifier",40    trust_remote_code=True,41    use_memory_efficient_attention=False)42 43web_page = """http://www.example.com44 45How to build a computer from scratch? Here are the components you need..."""46 47inputs = tokenizer([web_page], return_tensors="pt")48outputs = model(**inputs)49 50probs = outputs.logits.softmax(dim=-1)51print(probs.argmax(dim=-1))52# -> 5 ("Hardware" topic)53```54 55You can convert the `logits` of the model with a softmax to obtain a probability distribution over the following 24 categories (in order of labels, also see `id2label` and `label2id` in the model config):561. Adult572. Art & Design583. Software Dev.594. Crime & Law605. Education & Jobs616. Hardware627. Entertainment638. Social Life649. Fashion & Beauty6510. Finance & Business6611. Food & Dining6712. Games6813. Health6914. History7015. Home & Hobbies7116. Industrial7217. Literature7318. Politics7419. Religion7520. Science & Tech.7621. Software7722. Sports & Fitness7823. Transportation7924. Travel80 81The full definitions of the categories can be found in the [taxonomy config](https://github.com/CodeCreator/WebOrganizer/blob/main/define_domains/taxonomies/topics.yaml).82 83#### Efficient Inference84We recommend that you use the efficient gte-base-en-v1.5 implementation by enabling unpadding and memory efficient attention. This __requires installing `xformers`__ (see more [here](https://huggingface.co/Alibaba-NLP/new-impl#recommendation-enable-unpadding-and-acceleration-with-xformers)) and loading the model like:85```python86AutoModelForSequenceClassification.from_pretrained(87    "WebOrganizer/TopicClassifier",88    trust_remote_code=True,89    unpad_inputs=True,90    use_memory_efficient_attention=True,91    torch_dtype=torch.bfloat1692)93```94 95## Citation96```bibtex97@article{wettig2025organize,98  title={Organize the Web: Constructing Domains Enhances Pre-Training Data Curation},99  author={Alexander Wettig and Kyle Lo and Sewon Min and Hannaneh Hajishirzi and Danqi Chen and Luca Soldaini},100  journal={arXiv preprint arXiv:2502.10341},101  year={2025}102}103```