dewdev/dynamic_tinybert

1---2tags:3- question-answering4- bert5license: apache-2.06datasets:7- squad8language:9- en10model-index:11- name: dynamic-tinybert12  results: 13  - task:14      type: question-answering15      name: question-answering16    metrics:17    - type: f118      value: 88.7119 20---21 22## Model Details: Dynamic-TinyBERT: Boost TinyBERT's Inference Efficiency by Dynamic Sequence Length23 24Dynamic-TinyBERT has been fine-tuned for the NLP task of question answering, trained on the SQuAD 1.1 dataset. [Guskin et al. (2021)](https://neurips2021-nlp.github.io/papers/16/CameraReady/Dynamic_TinyBERT_NLSP2021_camera_ready.pdf) note:25 26> Dynamic-TinyBERT is a TinyBERT model that utilizes sequence-length reduction and Hyperparameter Optimization for enhanced inference efficiency per any computational budget. Dynamic-TinyBERT is trained only once, performing on-par with BERT and achieving an accuracy-speedup trade-off superior to any other efficient approaches (up to 3.3x with <1% loss-drop).27 28 29 30| Model Detail | Description |31| ----------- | ----------- | 32| Model Authors - Company | Intel | 33| Model Card Authors | Intel in collaboration with Hugging Face | 34| Date | November 22, 2021 | 35| Version | 1 | 36| Type | NLP - Question Answering | 37| Architecture | "For our Dynamic-TinyBERT model we use the architecture of TinyBERT6L: a small BERT model with 6 layers, a hidden size of 768, a feed forward size of 3072 and 12 heads." [Guskin et al. (2021)](https://gyuwankim.github.io/publication/dynamic-tinybert/poster.pdf) |38| Paper or Other Resources | [Paper](https://neurips2021-nlp.github.io/papers/16/CameraReady/Dynamic_TinyBERT_NLSP2021_camera_ready.pdf); [Poster](https://gyuwankim.github.io/publication/dynamic-tinybert/poster.pdf); [GitHub Repo](https://github.com/IntelLabs/Model-Compression-Research-Package) | 39| License | Apache 2.0 |40| Questions or Comments | [Community Tab](https://huggingface.co/Intel/dynamic_tinybert/discussions) and [Intel Developers Discord](https://discord.gg/rv2Gp55UJQ)|41 42| Intended Use | Description |43| ----------- | ----------- | 44| Primary intended uses | You can use the model for the NLP task of question answering: given a corpus of text, you can ask it a question about that text, and it will find the answer in the text. | 45| Primary intended users | Anyone doing question answering | 46| Out-of-scope uses |  The model should not be used to intentionally create hostile or alienating environments for people.|47 48### How to use49 50Here is how to import this model in Python:51 52 <details>53<summary> Click to expand </summary>54 55```python56import torch57from transformers import AutoTokenizer, AutoModelForQuestionAnswering58 59tokenizer = AutoTokenizer.from_pretrained("Intel/dynamic_tinybert")60model = AutoModelForQuestionAnswering.from_pretrained("Intel/dynamic_tinybert")61 62context = "remember the number 123456, I'll ask you later."63question = "What is the number I told you?"64 65# Tokenize the context and question66tokens = tokenizer.encode_plus(question, context, return_tensors="pt", truncation=True)67 68# Get the input IDs and attention mask69input_ids = tokens["input_ids"]70attention_mask = tokens["attention_mask"]71 72# Perform question answering73outputs = model(input_ids, attention_mask=attention_mask)74start_scores = outputs.start_logits75end_scores = outputs.end_logits76 77# Find the start and end positions of the answer78answer_start = torch.argmax(start_scores)79answer_end = torch.argmax(end_scores) + 180answer = tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(input_ids[0][answer_start:answer_end]))81 82# Print the answer83print("Answer:", answer)84 ```85</details>86 87 88| Factors | Description | 89| ----------- | ----------- | 90| Groups | Many Wikipedia articles with question and answer labels are contained in the training data | 91| Instrumentation | - |92| Environment | Training was completed on a Titan GPU. |93| Card Prompts | Model deployment on alternate hardware and software will change model performance |94 95| Metrics | Description | 96| ----------- | ----------- | 97| Model performance measures | F1 |98| Decision thresholds | - | 99| Approaches to uncertainty and variability | - | 100 101| Training and Evaluation Data | Description | 102| ----------- | ----------- | 103| Datasets | SQuAD1.1: "Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage, or the question might be unanswerable." (https://huggingface.co/datasets/squad)|104| Motivation | To build an efficient and accurate model for the question answering task. |105| Preprocessing | "We start with a pre-trained general-TinyBERT student, which was trained to learn the general knowledge of BERT using the general-distillation method presented by TinyBERT. We perform transformer distillation from a fine- tuned BERT teacher to the student, following the same training steps used in the original TinyBERT: (1) intermediate-layer distillation (ID) — learning the knowledge residing in the hidden states and attentions matrices, and (2) prediction-layer distillation (PD) — fitting the predictions of the teacher." ([Guskin et al., 2021](https://neurips2021-nlp.github.io/papers/16/CameraReady/Dynamic_TinyBERT_NLSP2021_camera_ready.pdf))| 106 107Model Performance Analysis:108 109| Model            | Max F1 (full model) | Best Speedup within BERT-1% |110|------------------|---------------------|-----------------------------|111| Dynamic-TinyBERT | 88.71               | 3.3x                        |112 113| Ethical Considerations | Description | 114| ----------- | ----------- | 115| Data | The training data come from Wikipedia articles |116| Human life | The model is not intended to inform decisions central to human life or flourishing. It is an aggregated set of labelled Wikipedia articles. | 117| Mitigations | No additional risk mitigation strategies were considered during model development. |118| Risks and harms | Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al., 2021](https://aclanthology.org/2021.acl-long.330.pdf), and [Bender et al., 2021](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)). Predictions generated by the model may include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups. Beyond this, the extent of the risks involved by using the model remain unknown.|119| Use cases | - | 120 121 122| Caveats and Recommendations |123| ----------- | 124| Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. There are no additional caveats or recommendations for this model. |125 126 127### BibTeX entry and citation info128```bibtex129@misc{https://doi.org/10.48550/arxiv.2111.09645,130  doi = {10.48550/ARXIV.2111.09645},131  132  url = {https://arxiv.org/abs/2111.09645},133  134  author = {Guskin, Shira and Wasserblat, Moshe and Ding, Ke and Kim, Gyuwan},135  136  keywords = {Computation and Language (cs.CL), Machine Learning (cs.LG), FOS: Computer and information sciences, FOS: Computer and information sciences},137  138  title = {Dynamic-TinyBERT: Boost TinyBERT's Inference Efficiency by Dynamic Sequence Length},139  140  publisher = {arXiv},141  142  year = {2021},143```