WebOrganizer/TopicClassifier-NoURL

1---2library_name: transformers3datasets:4- WebOrganizer/TopicAnnotations-Llama-3.1-8B5- WebOrganizer/TopicAnnotations-Llama-3.1-405B-FP86base_model:7- Alibaba-NLP/gte-base-en-v1.58---9# WebOrganizer/TopicClassifier-NoURL10 11[[Paper](https://arxiv.org/abs/2502.10341)] [[Website](https://weborganizer.allenai.org)] [[GitHub](https://github.com/CodeCreator/WebOrganizer)]12 13The TopicClassifier-NoURL organizes web content into 17 categories based on the text contents of web pages (without using URL information).14The model is a [gte-base-en-v1.5](https://huggingface.co/Alibaba-NLP/gte-base-en-v1.5) with 140M parameters fine-tuned on the following training data:151. [WebOrganizer/TopicAnnotations-Llama-3.1-8B](https://huggingface.co/datasets/WebOrganizer/TopicAnnotations-Llama-3.1-8B): 1M documents annotated by Llama-3.1-8B (first-stage training)162. [WebOrganizer/TopicAnnotations-Llama-3.1-405B-FP8](https://huggingface.co/datasets/WebOrganizer/TopicAnnotations-Llama-3.1-405B-FP8): 100K documents annotated by Llama-3.1-405B-FP8 (second-stage training)17 18#### All Domain Classifiers19- [WebOrganizer/FormatClassifier](https://huggingface.co/WebOrganizer/FormatClassifier)20- [WebOrganizer/FormatClassifier-NoURL](https://huggingface.co/WebOrganizer/FormatClassifier-NoURL)21- [WebOrganizer/TopicClassifier](https://huggingface.co/WebOrganizer/TopicClassifier)22- [WebOrganizer/TopicClassifier-NoURL](https://huggingface.co/WebOrganizer/TopicClassifier-NoURL) *← you are here!*23 24## Usage25 26This classifier expects input in the following format:27```28{text}29```30 31Example:32```python33from transformers import AutoTokenizer, AutoModelForSequenceClassification34 35tokenizer = AutoTokenizer.from_pretrained("WebOrganizer/TopicClassifier-NoURL")36model = AutoModelForSequenceClassification.from_pretrained(37    "WebOrganizer/TopicClassifier-NoURL",38    trust_remote_code=True,39    use_memory_efficient_attention=False)40 41web_page = """How to build a computer from scratch? Here are the components you need..."""42 43inputs = tokenizer([web_page], return_tensors="pt")44outputs = model(**inputs)45 46probs = outputs.logits.softmax(dim=-1)47print(probs.argmax(dim=-1))48# -> 5 ("Hardware" topic)49```50 51You can convert the `logits` of the model with a softmax to obtain a probability distribution over the following 24 categories (in order of labels, also see `id2label` and `label2id` in the model config):521. Adult532. Art & Design543. Software Dev.554. Crime & Law565. Education & Jobs576. Hardware587. Entertainment598. Social Life609. Fashion & Beauty6110. Finance & Business6211. Food & Dining6312. Games6413. Health6514. History6615. Home & Hobbies6716. Industrial6817. Literature6918. Politics7019. Religion7120. Science & Tech.7221. Software7322. Sports & Fitness7423. Transportation7524. Travel76 77The full definitions of the categories can be found in the [taxonomy config](https://github.com/CodeCreator/WebOrganizer/blob/main/define_domains/taxonomies/topics.yaml).78 79#### Efficient Inference80We recommend that you use the efficient gte-base-en-v1.5 implementation by enabling unpadding and memory efficient attention. This __requires installing `xformers`__ (see more [here](https://huggingface.co/Alibaba-NLP/new-impl#recommendation-enable-unpadding-and-acceleration-with-xformers)) and loading the model like:81```python82AutoModelForSequenceClassification.from_pretrained(83    "WebOrganizer/TopicClassifier-NoURL",84    trust_remote_code=True,85    unpad_inputs=True,86    use_memory_efficient_attention=True,87    torch_dtype=torch.bfloat1688)89```90 91## Citation92```bibtex93@article{wettig2025organize,94  title={Organize the Web: Constructing Domains Enhances Pre-Training Data Curation},95  author={Alexander Wettig and Kyle Lo and Sewon Min and Hannaneh Hajishirzi and Danqi Chen and Luca Soldaini},96  journal={arXiv preprint arXiv:2502.10341},97  year={2025}98}99```