chendl/compositional_test
1
1<!---2Copyright 2020 The HuggingFace Team. All rights reserved.3Licensed under the Apache License, Version 2.0 (the "License");4you may not use this file except in compliance with the License.5You may obtain a copy of the License at6 7 http://www.apache.org/licenses/LICENSE-2.08 9Unless required by applicable law or agreed to in writing, software10distributed under the License is distributed on an "AS IS" BASIS,11WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.12See the License for the specific language governing permissions and13limitations under the License.14-->15 16# Examples17 18This folder contains actively maintained examples of use of 🤗 Transformers using the PyTorch backend, organized by ML task.19 20## The Big Table of Tasks21 22Here is the list of all our examples:23- with information on whether they are **built on top of `Trainer`** (if not, they still work, they might24 just lack some features),25- whether or not they have a version using the [🤗 Accelerate](https://github.com/huggingface/accelerate) library.26- whether or not they leverage the [🤗 Datasets](https://github.com/huggingface/datasets) library.27- links to **Colab notebooks** to walk through the scripts and run them easily,28<!--29Coming soon!30- links to **Cloud deployments** to be able to deploy large-scale trainings in the Cloud with little to no setup.31-->32 33| Task | Example datasets | Trainer support | 🤗 Accelerate | 🤗 Datasets | Colab34|---|---|:---:|:---:|:---:|:---:|35| [**`language-modeling`**](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling) | [WikiText-2](https://huggingface.co/datasets/wikitext) | ✅ | ✅ | ✅ | [](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb)36| [**`multiple-choice`**](https://github.com/huggingface/transformers/tree/main/examples/pytorch/multiple-choice) | [SWAG](https://huggingface.co/datasets/swag) | ✅ | ✅ | ✅ | [](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice.ipynb)37| [**`question-answering`**](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering) | [SQuAD](https://huggingface.co/datasets/squad) | ✅ | ✅ | ✅ | [](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb)38| [**`summarization`**](https://github.com/huggingface/transformers/tree/main/examples/pytorch/summarization) | [XSum](https://huggingface.co/datasets/xsum) | ✅ | ✅ | ✅ | [](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/summarization.ipynb)39| [**`text-classification`**](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification) | [GLUE](https://huggingface.co/datasets/glue) | ✅ | ✅ | ✅ | [](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification.ipynb)40| [**`text-generation`**](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-generation) | - | n/a | - | - | [](https://colab.research.google.com/github/huggingface/blog/blob/main/notebooks/02_how_to_generate.ipynb)41| [**`token-classification`**](https://github.com/huggingface/transformers/tree/main/examples/pytorch/token-classification) | [CoNLL NER](https://huggingface.co/datasets/conll2003) | ✅ |✅ | ✅ | [](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification.ipynb)42| [**`translation`**](https://github.com/huggingface/transformers/tree/main/examples/pytorch/translation) | [WMT](https://huggingface.co/datasets/wmt17) | ✅ | ✅ |✅ | [](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/translation.ipynb)43| [**`speech-recognition`**](https://github.com/huggingface/transformers/tree/main/examples/pytorch/speech-recognition) | [TIMIT](https://huggingface.co/datasets/timit_asr) | ✅ | - |✅ | [](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/speech_recognition.ipynb)44| [**`multi-lingual speech-recognition`**](https://github.com/huggingface/transformers/tree/main/examples/pytorch/speech-recognition) | [Common Voice](https://huggingface.co/datasets/common_voice) | ✅ | - |✅ | [](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multi_lingual_speech_recognition.ipynb)45| [**`audio-classification`**](https://github.com/huggingface/transformers/tree/main/examples/pytorch/audio-classification) | [SUPERB KS](https://huggingface.co/datasets/superb) | ✅ | - |✅ | [](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/audio_classification.ipynb)46| [**`image-pretraining`**](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-pretraining) | [ImageNet-1k](https://huggingface.co/datasets/imagenet-1k) | ✅ | - |✅ | /47| [**`image-classification`**](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) | [CIFAR-10](https://huggingface.co/datasets/cifar10) | ✅ | ✅ |✅ | [](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb)48| [**`semantic-segmentation`**](https://github.com/huggingface/transformers/tree/main/examples/pytorch/semantic-segmentation) | [SCENE_PARSE_150](https://huggingface.co/datasets/scene_parse_150) | ✅ | ✅ |✅ | [](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/semantic_segmentation.ipynb)49 50 51## Running quick tests52 53Most examples are equipped with a mechanism to truncate the number of dataset samples to the desired length. This is useful for debugging purposes, for example to quickly check that all stages of the programs can complete, before running the same setup on the full dataset which may take hours to complete.54 55For example here is how to truncate all three splits to just 50 samples each:56```57examples/pytorch/token-classification/run_ner.py \58--max_train_samples 50 \59--max_eval_samples 50 \60--max_predict_samples 50 \61[...]62```63 64Most example scripts should have the first two command line arguments and some have the third one. You can quickly check if a given example supports any of these by passing a `-h` option, e.g.:65```66examples/pytorch/token-classification/run_ner.py -h67```68 69## Resuming training70 71You can resume training from a previous checkpoint like this:72 731. Pass `--output_dir previous_output_dir` without `--overwrite_output_dir` to resume training from the latest checkpoint in `output_dir` (what you would use if the training was interrupted, for instance).742. Pass `--resume_from_checkpoint path_to_a_specific_checkpoint` to resume training from that checkpoint folder.75 76Should you want to turn an example into a notebook where you'd no longer have access to the command77line, 🤗 Trainer supports resuming from a checkpoint via `trainer.train(resume_from_checkpoint)`.78 791. If `resume_from_checkpoint` is `True` it will look for the last checkpoint in the value of `output_dir` passed via `TrainingArguments`.802. If `resume_from_checkpoint` is a path to a specific checkpoint it will use that saved checkpoint folder to resume the training from.81 82 83### Upload the trained/fine-tuned model to the Hub84 85All the example scripts support automatic upload of your final model to the [Model Hub](https://huggingface.co/models) by adding a `--push_to_hub` argument. It will then create a repository with your username slash the name of the folder you are using as `output_dir`. For instance, `"sgugger/test-mrpc"` if your username is `sgugger` and you are working in the folder `~/tmp/test-mrpc`.86 87To specify a given repository name, use the `--hub_model_id` argument. You will need to specify the whole repository name (including your username), for instance `--hub_model_id sgugger/finetuned-bert-mrpc`. To upload to an organization you are a member of, just use the name of that organization instead of your username: `--hub_model_id huggingface/finetuned-bert-mrpc`.88 89A few notes on this integration:90 91- you will need to be logged in to the Hugging Face website locally for it to work, the easiest way to achieve this is to run `huggingface-cli login` and then type your username and password when prompted. You can also pass along your authentication token with the `--hub_token` argument.92- the `output_dir` you pick will either need to be a new folder or a local clone of the distant repository you are using.93 94## Distributed training and mixed precision95 96All the PyTorch scripts mentioned above work out of the box with distributed training and mixed precision, thanks to97the [Trainer API](https://huggingface.co/transformers/main_classes/trainer.html). To launch one of them on _n_ GPUs,98use the following command:99 100```bash101python -m torch.distributed.launch \102 --nproc_per_node number_of_gpu_you_have path_to_script.py \103 --all_arguments_of_the_script104```105 106As an example, here is how you would fine-tune the BERT large model (with whole word masking) on the text107classification MNLI task using the `run_glue` script, with 8 GPUs:108 109```bash110python -m torch.distributed.launch \111 --nproc_per_node 8 pytorch/text-classification/run_glue.py \112 --model_name_or_path bert-large-uncased-whole-word-masking \113 --task_name mnli \114 --do_train \115 --do_eval \116 --max_seq_length 128 \117 --per_device_train_batch_size 8 \118 --learning_rate 2e-5 \119 --num_train_epochs 3.0 \120 --output_dir /tmp/mnli_output/121```122 123If you have a GPU with mixed precision capabilities (architecture Pascal or more recent), you can use mixed precision124training with PyTorch 1.6.0 or latest, or by installing the [Apex](https://github.com/NVIDIA/apex) library for previous125versions. Just add the flag `--fp16` to your command launching one of the scripts mentioned above!126 127Using mixed precision training usually results in 2x-speedup for training with the same final results (as shown in128[this table](https://github.com/huggingface/transformers/tree/main/examples/text-classification#mixed-precision-training)129for text classification).130 131## Running on TPUs132 133When using Tensorflow, TPUs are supported out of the box as a `tf.distribute.Strategy`.134 135When using PyTorch, we support TPUs thanks to `pytorch/xla`. For more context and information on how to setup your TPU environment refer to Google's documentation and to the136very detailed [pytorch/xla README](https://github.com/pytorch/xla/blob/master/README.md).137 138In this repo, we provide a very simple launcher script named139[xla_spawn.py](https://github.com/huggingface/transformers/tree/main/examples/pytorch/xla_spawn.py) that lets you run our140example scripts on multiple TPU cores without any boilerplate. Just pass a `--num_cores` flag to this script, then your141regular training script with its arguments (this is similar to the `torch.distributed.launch` helper for142`torch.distributed`):143 144```bash145python xla_spawn.py --num_cores num_tpu_you_have \146 path_to_script.py \147 --all_arguments_of_the_script148```149 150As an example, here is how you would fine-tune the BERT large model (with whole word masking) on the text151classification MNLI task using the `run_glue` script, with 8 TPUs (from this folder):152 153```bash154python xla_spawn.py --num_cores 8 \155 text-classification/run_glue.py \156 --model_name_or_path bert-large-uncased-whole-word-masking \157 --task_name mnli \158 --do_train \159 --do_eval \160 --max_seq_length 128 \161 --per_device_train_batch_size 8 \162 --learning_rate 2e-5 \163 --num_train_epochs 3.0 \164 --output_dir /tmp/mnli_output/165```166 167## Using Accelerate168 169Most PyTorch example scripts have a version using the [🤗 Accelerate](https://github.com/huggingface/accelerate) library170that exposes the training loop so it's easy for you to customize or tweak them to your needs. They all require you to171install `accelerate` with the latest development version172 173```bash174pip install git+https://github.com/huggingface/accelerate175```176 177Then you can easily launch any of the scripts by running178 179```bash180accelerate config181```182 183and reply to the questions asked. Then184 185```bash186accelerate test187```188 189that will check everything is ready for training. Finally, you can launch training with190 191```bash192accelerate launch path_to_script.py --args_to_script193```194 195## Logging & Experiment tracking196 197You can easily log and monitor your runs code. The following are currently supported:198 199* [TensorBoard](https://www.tensorflow.org/tensorboard)200* [Weights & Biases](https://docs.wandb.ai/integrations/huggingface)201* [Comet ML](https://www.comet.ml/docs/python-sdk/huggingface/)202* [Neptune](https://docs.neptune.ai/integrations-and-supported-tools/model-training/hugging-face)203* [ClearML](https://clear.ml/docs/latest/docs/getting_started/ds/ds_first_steps)204 205### Weights & Biases206 207To use Weights & Biases, install the wandb package with:208 209```bash210pip install wandb211```212 213Then log in the command line:214 215```bash216wandb login217```218 219If you are in Jupyter or Colab, you should login with:220 221```python222import wandb223wandb.login()224```225 226To enable logging to W&B, include `"wandb"` in the `report_to` of your `TrainingArguments` or script. Or just pass along `--report_to all` if you have `wandb` installed.227 228Whenever you use `Trainer` or `TFTrainer` classes, your losses, evaluation metrics, model topology and gradients (for `Trainer` only) will automatically be logged.229 230Advanced configuration is possible by setting environment variables:231 232| Environment Variable | Value |233|---|---|234| WANDB_LOG_MODEL | Log the model as artifact (log the model as artifact at the end of training) (`false` by default) |235| WANDB_WATCH | one of `gradients` (default) to log histograms of gradients, `all` to log histograms of both gradients and parameters, or `false` for no histogram logging |236| WANDB_PROJECT | Organize runs by project |237 238Set run names with `run_name` argument present in scripts or as part of `TrainingArguments`.239 240Additional configuration options are available through generic [wandb environment variables](https://docs.wandb.com/library/environment-variables).241 242Refer to related [documentation & examples](https://docs.wandb.ai/integrations/huggingface).243 244### Comet.ml245 246To use `comet_ml`, install the Python package with:247 248```bash249pip install comet_ml250```251 252or if in a Conda environment:253 254```bash255conda install -c comet_ml -c anaconda -c conda-forge comet_ml256```257 258### Neptune259 260First, install the Neptune client library. You can do it with either `pip` or `conda`:261 262`pip`:263 264```bash265pip install neptune266```267 268`conda`:269 270```bash271conda install -c conda-forge neptune272```273 274Next, in your model training script, import `NeptuneCallback`:275 276```python277from transformers.integrations import NeptuneCallback278```279 280To enable Neptune logging, in your `TrainingArguments`, set the `report_to` argument to `"neptune"`:281 282```python283training_args = TrainingArguments(284 "quick-training-distilbert-mrpc", 285 evaluation_strategy="steps",286 eval_steps=20,287 report_to="neptune",288)289 290trainer = Trainer(291 model,292 training_args,293 ...294)295```296 297**Note:** This method requires saving your Neptune credentials as environment variables (see the bottom of the section).298 299Alternatively, for more logging options, create a Neptune callback:300 301```python302neptune_callback = NeptuneCallback()303```304 305To add more detail to the tracked run, you can supply optional arguments to `NeptuneCallback`.306 307Some examples:308 309```python310neptune_callback = NeptuneCallback(311 name = "DistilBERT",312 description = "DistilBERT fine-tuned on GLUE/MRPC",313 tags = ["args-callback", "fine-tune", "MRPC"], # tags help you manage runs in Neptune314 base_namespace="callback", # the default is "finetuning"315 log_checkpoints = "best", # other options are "last", "same", and None316 capture_hardware_metrics = False, # additional keyword arguments for a Neptune run317)318```319 320Pass the callback to the Trainer:321 322```python323training_args = TrainingArguments(..., report_to=None)324trainer = Trainer(325 model,326 training_args,327 ...328 callbacks=[neptune_callback],329)330```331 332Now, when you start the training with `trainer.train()`, your metadata will be logged in Neptune.333 334**Note:** Although you can pass your **Neptune API token** and **project name** as arguments when creating the callback, the recommended way is to save them as environment variables:335 336| Environment variable | Value |337| :------------------- | :--------------------------------------------------- |338| `NEPTUNE_API_TOKEN` | Your Neptune API token. To find and copy it, click your Neptune avatar and select **Get your API token**. |339| `NEPTUNE_PROJECT` | The full name of your Neptune project (`workspace-name/project-name`). To find and copy it, head to **project settings** → **Properties**. |340 341For detailed instructions and examples, see the [Neptune docs](https://docs.neptune.ai/integrations/transformers/).342 343### ClearML344 345To use ClearML, install the clearml package with:346 347```bash348pip install clearml349```350 351Then [create new credentials]() from the ClearML Server. You can get a free hosted server [here]() or [self-host your own]()!352After creating your new credentials, you can either copy the local snippet which you can paste after running:353 354```bash355clearml-init356```357 358Or you can copy the jupyter snippet if you are in Jupyter or Colab:359 360```python361%env CLEARML_WEB_HOST=https://app.clear.ml362%env CLEARML_API_HOST=https://api.clear.ml363%env CLEARML_FILES_HOST=https://files.clear.ml364%env CLEARML_API_ACCESS_KEY=***365%env CLEARML_API_SECRET_KEY=***366```367 368 369To enable logging to ClearML, include `"clearml"` in the `report_to` of your `TrainingArguments` or script. Or just pass along `--report_to all` if you have `clearml` already installed.370 371Advanced configuration is possible by setting environment variables:372 373| Environment Variable | Value |374|---|---|375| CLEARML_PROJECT | Name of the project in ClearML. (default: `"HuggingFace Transformers"`) |376| CLEARML_TASK | Name of the task in ClearML. (default: `"Trainer"`) |377 378Additional configuration options are available through generic [clearml environment variables](https://clear.ml/docs/latest/docs/configs/env_vars).379