DAMO-NLP-SG/zero-shot-classify-SSTuning-large

1---2inference: false3license: mit4tags:5- Zero-Shot Classification6pipeline_tag: zero-shot-classification7---8# Zero-shot text classification (large-sized model) trained with self-supervised tuning9 10Zero-shot text classification model trained with self-supervised tuning (SSTuning). 11It was introduced in the paper [Zero-Shot Text Classification via Self-Supervised Tuning](https://arxiv.org/abs/2305.11442) by 12Chaoqun Liu, Wenxuan Zhang, Guizhen Chen, Xiaobao Wu, Anh Tuan Luu, Chip Hong Chang, Lidong Bing13and first released in [this repository](https://github.com/DAMO-NLP-SG/SSTuning).14 15The model backbone is RoBERTa-large.16 17## Model description18The model is tuned with unlabeled data using a learning objective called first sentence prediction (FSP). 19The FSP task is designed by considering both the nature of the unlabeled corpus and the input/output format of classification tasks. 20The training and validation sets are constructed from the unlabeled corpus using FSP. 21 22During tuning, BERT-like pre-trained masked language 23models such as RoBERTa and ALBERT are employed as the backbone, and an output layer for classification is added. 24The learning objective for FSP is to predict the index of the correct label. 25A cross-entropy loss is used for tuning the model.26 27## Model variations28There are three versions of models released. The details are: 29 30| Model | Backbone | #params | accuracy | Speed | #Training data31|------------|-----------|----------|-------|-------|----|32|   [zero-shot-classify-SSTuning-base](https://huggingface.co/DAMO-NLP-SG/zero-shot-classify-SSTuning-base)    |  [roberta-base](https://huggingface.co/roberta-base)      |  125M    |  Low    |  High    | 20.48M |  33|   [zero-shot-classify-SSTuning-large](https://huggingface.co/DAMO-NLP-SG/zero-shot-classify-SSTuning-large)    |    [roberta-large](https://huggingface.co/roberta-large)      | 355M     |   Medium   | Medium | 5.12M |34|   [zero-shot-classify-SSTuning-ALBERT](https://huggingface.co/DAMO-NLP-SG/zero-shot-classify-SSTuning-ALBERT)   |  [albert-xxlarge-v2](https://huggingface.co/albert-xxlarge-v2)      |  235M   |    High  | Low| 5.12M |35 36Please note that zero-shot-classify-SSTuning-base is trained with more data (20.48M) than the paper, as this will increase the accuracy.37 38 39## Intended uses & limitations40The model can be used for zero-shot text classification such as sentiment analysis and topic classification. No further finetuning is needed.41 42The number of labels should be 2 ~ 20. 43 44### How to use45You can try the model with the Colab [Notebook](https://colab.research.google.com/drive/17bqc8cXFF-wDmZ0o8j7sbrQB9Cq7Gowr?usp=sharing).46 47```python48from transformers import AutoTokenizer, AutoModelForSequenceClassification49import torch, string, random50 51tokenizer = AutoTokenizer.from_pretrained("DAMO-NLP-SG/zero-shot-classify-SSTuning-large")52model = AutoModelForSequenceClassification.from_pretrained("DAMO-NLP-SG/zero-shot-classify-SSTuning-large")53 54text = "I love this place! The food is always so fresh and delicious."55list_label = ["negative", "positive"]56 57device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')58list_ABC = [x for x in string.ascii_uppercase]59 60def check_text(model, text, list_label, shuffle=False): 61    list_label = [x+'.' if x[-1] != '.' else x for x in list_label]62    list_label_new = list_label + [tokenizer.pad_token]* (20 - len(list_label))63    if shuffle: 64        random.shuffle(list_label_new)65    s_option = ' '.join(['('+list_ABC[i]+') '+list_label_new[i] for i in range(len(list_label_new))])66    text = f'{s_option} {tokenizer.sep_token} {text}'67 68    model.to(device).eval()69    encoding = tokenizer([text],truncation=True, max_length=512,return_tensors='pt')70    item = {key: val.to(device) for key, val in encoding.items()}71    logits = model(**item).logits72    73    logits = logits if shuffle else logits[:,0:len(list_label)]74    probs = torch.nn.functional.softmax(logits, dim = -1).tolist()75    predictions = torch.argmax(logits, dim=-1).item() 76    probabilities = [round(x,5) for x in probs[0]]77 78    print(f'prediction:    {predictions} => ({list_ABC[predictions]}) {list_label_new[predictions]}')79    print(f'probability:   {round(probabilities[predictions]*100,2)}%')80 81check_text(model, text, list_label)82# prediction:    1 => (B) positive.83# probability:   99.84%84```85 86 87### BibTeX entry and citation info88```bibtxt89@inproceedings{acl23/SSTuning,90  author    = {Chaoqun Liu and91               Wenxuan Zhang and92               Guizhen Chen and93               Xiaobao Wu and94               Anh Tuan Luu and95               Chip Hong Chang and 96               Lidong Bing},97  title     = {Zero-Shot Text Classification via Self-Supervised Tuning},98  booktitle = {Findings of the Association for Computational Linguistics: ACL 2023},99  year      = {2023},100  url       = {https://arxiv.org/abs/2305.11442},101}102```