chendl/compositional_test

1## Token classification2 3Based on the scripts [`run_ner.py`](https://github.com/huggingface/transformers/blob/main/examples/legacy/token-classification/run_ner.py).4 5The following examples are covered in this section:6 7* NER on the GermEval 2014 (German NER) dataset8* Emerging and Rare Entities task: WNUT’17 (English NER) dataset9 10Details and results for the fine-tuning provided by @stefan-it.11 12### GermEval 2014 (German NER) dataset13 14#### Data (Download and pre-processing steps)15 16Data can be obtained from the [GermEval 2014](https://sites.google.com/site/germeval2014ner/data) shared task page.17 18Here are the commands for downloading and pre-processing train, dev and test datasets. The original data format has four (tab-separated) columns, in a pre-processing step only the two relevant columns (token and outer span NER annotation) are extracted:19 20```bash21curl -L 'https://drive.google.com/uc?export=download&id=1Jjhbal535VVz2ap4v4r_rN1UEHTdLK5P' \22| grep -v "^#" | cut -f 2,3 | tr '\t' ' ' > train.txt.tmp23curl -L 'https://drive.google.com/uc?export=download&id=1ZfRcQThdtAR5PPRjIDtrVP7BtXSCUBbm' \24| grep -v "^#" | cut -f 2,3 | tr '\t' ' ' > dev.txt.tmp25curl -L 'https://drive.google.com/uc?export=download&id=1u9mb7kNJHWQCWyweMDRMuTFoOHOfeBTH' \26| grep -v "^#" | cut -f 2,3 | tr '\t' ' ' > test.txt.tmp27```28 29The GermEval 2014 dataset contains some strange "control character" tokens like `'\x96', '\u200e', '\x95', '\xad' or '\x80'`.30One problem with these tokens is, that `BertTokenizer` returns an empty token for them, resulting in misaligned `InputExample`s.31The `preprocess.py` script located in the `scripts` folder a) filters these tokens and b) splits longer sentences into smaller ones (once the max. subtoken length is reached).32 33Let's define some variables that we need for further pre-processing steps and training the model:34 35```bash36export MAX_LENGTH=12837export BERT_MODEL=bert-base-multilingual-cased38```39 40Run the pre-processing script on training, dev and test datasets:41 42```bash43python3 scripts/preprocess.py train.txt.tmp $BERT_MODEL $MAX_LENGTH > train.txt44python3 scripts/preprocess.py dev.txt.tmp $BERT_MODEL $MAX_LENGTH > dev.txt45python3 scripts/preprocess.py test.txt.tmp $BERT_MODEL $MAX_LENGTH > test.txt46```47 48The GermEval 2014 dataset has much more labels than CoNLL-2002/2003 datasets, so an own set of labels must be used:49 50```bash51cat train.txt dev.txt test.txt | cut -d " " -f 2 | grep -v "^$"| sort | uniq > labels.txt52```53 54#### Prepare the run55 56Additional environment variables must be set:57 58```bash59export OUTPUT_DIR=germeval-model60export BATCH_SIZE=3261export NUM_EPOCHS=362export SAVE_STEPS=75063export SEED=164```65 66#### Run the Pytorch version67 68To start training, just run:69 70```bash71python3 run_ner.py --data_dir ./ \72--labels ./labels.txt \73--model_name_or_path $BERT_MODEL \74--output_dir $OUTPUT_DIR \75--max_seq_length  $MAX_LENGTH \76--num_train_epochs $NUM_EPOCHS \77--per_device_train_batch_size $BATCH_SIZE \78--save_steps $SAVE_STEPS \79--seed $SEED \80--do_train \81--do_eval \82--do_predict83```84 85If your GPU supports half-precision training, just add the `--fp16` flag. After training, the model will be both evaluated on development and test datasets.86 87#### JSON-based configuration file88 89Instead of passing all parameters via commandline arguments, the `run_ner.py` script also supports reading parameters from a json-based configuration file:90 91```json92{93    "data_dir": ".",94    "labels": "./labels.txt",95    "model_name_or_path": "bert-base-multilingual-cased",96    "output_dir": "germeval-model",97    "max_seq_length": 128,98    "num_train_epochs": 3,99    "per_device_train_batch_size": 32,100    "save_steps": 750,101    "seed": 1,102    "do_train": true,103    "do_eval": true,104    "do_predict": true105}106```107 108It must be saved with a `.json` extension and can be used by running `python3 run_ner.py config.json`.109 110#### Evaluation111 112Evaluation on development dataset outputs the following for our example:113 114```bash11510/04/2019 00:42:06 - INFO - __main__ -   ***** Eval results  *****11610/04/2019 00:42:06 - INFO - __main__ -     f1 = 0.862334801762114611710/04/2019 00:42:06 - INFO - __main__ -     loss = 0.0718386966697554311810/04/2019 00:42:06 - INFO - __main__ -     precision = 0.846791636625811111910/04/2019 00:42:06 - INFO - __main__ -     recall = 0.8784592370979806120```121 122On the test dataset the following results could be achieved:123 124```bash12510/04/2019 00:42:42 - INFO - __main__ -   ***** Eval results  *****12610/04/2019 00:42:42 - INFO - __main__ -     f1 = 0.861438965238480312710/04/2019 00:42:42 - INFO - __main__ -     loss = 0.0706460248745478212810/04/2019 00:42:42 - INFO - __main__ -     precision = 0.860465116279069712910/04/2019 00:42:42 - INFO - __main__ -     recall = 0.8624150210424085130```131 132#### Run the Tensorflow 2 version133 134To start training, just run:135 136```bash137python3 run_tf_ner.py --data_dir ./ \138--labels ./labels.txt \139--model_name_or_path $BERT_MODEL \140--output_dir $OUTPUT_DIR \141--max_seq_length  $MAX_LENGTH \142--num_train_epochs $NUM_EPOCHS \143--per_device_train_batch_size $BATCH_SIZE \144--save_steps $SAVE_STEPS \145--seed $SEED \146--do_train \147--do_eval \148--do_predict149```150 151Such as the Pytorch version, if your GPU supports half-precision training, just add the `--fp16` flag. After training, the model will be both evaluated on development and test datasets.152 153#### Evaluation154 155Evaluation on development dataset outputs the following for our example:156```bash157           precision    recall  f1-score   support158 159 LOCderiv     0.7619    0.6154    0.6809        52160  PERpart     0.8724    0.8997    0.8858      4057161  OTHpart     0.9360    0.9466    0.9413       711162  ORGpart     0.7015    0.6989    0.7002       269163  LOCpart     0.7668    0.8488    0.8057       496164      LOC     0.8745    0.9191    0.8963       235165 ORGderiv     0.7723    0.8571    0.8125        91166 OTHderiv     0.4800    0.6667    0.5581        18167      OTH     0.5789    0.6875    0.6286        16168 PERderiv     0.5385    0.3889    0.4516        18169      PER     0.5000    0.5000    0.5000         2170      ORG     0.0000    0.0000    0.0000         3171 172micro avg     0.8574    0.8862    0.8715      5968173macro avg     0.8575    0.8862    0.8713      5968174```175 176On the test dataset the following results could be achieved:177```bash178           precision    recall  f1-score   support179 180  PERpart     0.8847    0.8944    0.8896      9397181  OTHpart     0.9376    0.9353    0.9365      1639182  ORGpart     0.7307    0.7044    0.7173       697183      LOC     0.9133    0.9394    0.9262       561184  LOCpart     0.8058    0.8157    0.8107      1150185      ORG     0.0000    0.0000    0.0000         8186 OTHderiv     0.5882    0.4762    0.5263        42187 PERderiv     0.6571    0.5227    0.5823        44188      OTH     0.4906    0.6667    0.5652        39189 ORGderiv     0.7016    0.7791    0.7383       172190 LOCderiv     0.8256    0.6514    0.7282       109191      PER     0.0000    0.0000    0.0000        11192 193micro avg     0.8722    0.8774    0.8748     13869194macro avg     0.8712    0.8774    0.8740     13869195```196 197### Emerging and Rare Entities task: WNUT’17 (English NER) dataset198 199Description of the WNUT’17 task from the [shared task website](http://noisy-text.github.io/2017/index.html):200 201> The WNUT’17 shared task focuses on identifying unusual, previously-unseen entities in the context of emerging discussions.202> Named entities form the basis of many modern approaches to other tasks (like event clustering and summarization), but recall on203> them is a real problem in noisy text - even among annotators. This drop tends to be due to novel entities and surface forms.204 205Six labels are available in the dataset. An overview can be found on this [page](http://noisy-text.github.io/2017/files/).206 207#### Data (Download and pre-processing steps)208 209The dataset can be downloaded from the [official GitHub](https://github.com/leondz/emerging_entities_17) repository.210 211The following commands show how to prepare the dataset for fine-tuning:212 213```bash214mkdir -p data_wnut_17215 216curl -L 'https://github.com/leondz/emerging_entities_17/raw/master/wnut17train.conll'  | tr '\t' ' ' > data_wnut_17/train.txt.tmp217curl -L 'https://github.com/leondz/emerging_entities_17/raw/master/emerging.dev.conll' | tr '\t' ' ' > data_wnut_17/dev.txt.tmp218curl -L 'https://raw.githubusercontent.com/leondz/emerging_entities_17/master/emerging.test.annotated' | tr '\t' ' ' > data_wnut_17/test.txt.tmp219```220 221Let's define some variables that we need for further pre-processing steps:222 223```bash224export MAX_LENGTH=128225export BERT_MODEL=bert-large-cased226```227 228Here we use the English BERT large model for fine-tuning.229The `preprocess.py` scripts splits longer sentences into smaller ones (once the max. subtoken length is reached):230 231```bash232python3 scripts/preprocess.py data_wnut_17/train.txt.tmp $BERT_MODEL $MAX_LENGTH > data_wnut_17/train.txt233python3 scripts/preprocess.py data_wnut_17/dev.txt.tmp $BERT_MODEL $MAX_LENGTH > data_wnut_17/dev.txt234python3 scripts/preprocess.py data_wnut_17/test.txt.tmp $BERT_MODEL $MAX_LENGTH > data_wnut_17/test.txt235```236 237In the last pre-processing step, the `labels.txt` file needs to be generated. This file contains all available labels:238 239```bash240cat data_wnut_17/train.txt data_wnut_17/dev.txt data_wnut_17/test.txt | cut -d " " -f 2 | grep -v "^$"| sort | uniq > data_wnut_17/labels.txt241```242 243#### Run the Pytorch version244 245Fine-tuning with the PyTorch version can be started using the `run_ner.py` script. In this example we use a JSON-based configuration file.246 247This configuration file looks like:248 249```json250{251    "data_dir": "./data_wnut_17",252    "labels": "./data_wnut_17/labels.txt",253    "model_name_or_path": "bert-large-cased",254    "output_dir": "wnut-17-model-1",255    "max_seq_length": 128,256    "num_train_epochs": 3,257    "per_device_train_batch_size": 32,258    "save_steps": 425,259    "seed": 1,260    "do_train": true,261    "do_eval": true,262    "do_predict": true,263    "fp16": false264}265```266 267If your GPU supports half-precision training, please set `fp16` to `true`.268 269Save this JSON-based configuration under `wnut_17.json`. The fine-tuning can be started with `python3 run_ner_old.py wnut_17.json`.270 271#### Evaluation272 273Evaluation on development dataset outputs the following:274 275```bash27605/29/2020 23:33:44 - INFO - __main__ -   ***** Eval results *****27705/29/2020 23:33:44 - INFO - __main__ -     eval_loss = 0.2650523528621227527805/29/2020 23:33:44 - INFO - __main__ -     eval_precision = 0.700826446280991827905/29/2020 23:33:44 - INFO - __main__ -     eval_recall = 0.50717703349282328005/29/2020 23:33:44 - INFO - __main__ -     eval_f1 = 0.588480222068008428105/29/2020 23:33:44 - INFO - __main__ -     epoch = 3.0282```283 284On the test dataset the following results could be achieved:285 286```bash28705/29/2020 23:33:44 - INFO - transformers.trainer -   ***** Running Prediction *****28805/29/2020 23:34:02 - INFO - __main__ -     eval_loss = 0.3094880650097354728905/29/2020 23:34:02 - INFO - __main__ -     eval_precision = 0.584010840108401129005/29/2020 23:34:02 - INFO - __main__ -     eval_recall = 0.399443929564411529105/29/2020 23:34:02 - INFO - __main__ -     eval_f1 = 0.47440836543753434292```293 294WNUT’17 is a very difficult task. Current state-of-the-art results on this dataset can be found [here](https://nlpprogress.com/english/named_entity_recognition.html).295