R11484/source_code/transformers/examplesa8311170c72cmain
examples
README.md
<!--- Copyright 2020 The HuggingFace Team. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. -->
Examples
This folder contains actively maintained examples of use of ๐ค Transformers organized along NLP tasks. If you are looking for an example that used to be in this folder, it may have moved to our research projects subfolder (which contains frozen snapshots of research projects).
Important note
Important
To make sure you can successfully run the latest versions of the example scripts, you have to install the library from source and install some example-specific requirements. To do this, execute the following steps in a new virtual environment:
bash git clone https://github.com/huggingface/transformers cd transformers pip install .
Then cd in the example folder of your choice and run
bash pip install -r requirements.txt
Alternatively, you can run the version of the examples as they were for your current version of Transformers via (for instance with v3.5.1):
bash git checkout tags/v3.5.1
The Big Table of Tasks
Here is the list of all our examples:
- with information on whether they are built on top of Trainer/TFTrainer (if not, they still work, they might just lack some features),
- whether or not they leverage the ๐ค Datasets library.
- links to Colab notebooks to walk through the scripts and run them easily,
<!-- Coming soon!
- links to Cloud deployments to be able to deploy large-scale trainings in the Cloud with little to no setup.
-->
<!--
One-click Deploy to Cloud (wip)
Coming soon! -->
Distributed training and mixed precision
All the PyTorch scripts mentioned above work out of the box with distributed training and mixed precision, thanks to the Trainer API. To launch one of them on _n_ GPUS, use the following command:
bash python -m torch.distributed.launch \ --nproc_per_node number_of_gpu_you_have path_to_script.py \ --all_arguments_of_the_script
As an example, here is how you would fine-tune the BERT large model (with whole word masking) on the text classification MNLI task using the run_glue script, with 8 GPUs:
bash python -m torch.distributed.launch \ --nproc_per_node 8 text-classification/run_glue.py \ --model_name_or_path bert-large-uncased-whole-word-masking \ --task_name mnli \ --do_train \ --do_eval \ --max_seq_length 128 \ --per_device_train_batch_size 8 \ --learning_rate 2e-5 \ --num_train_epochs 3.0 \ --output_dir /tmp/mnli_output/
If you have a GPU with mixed precision capabilities (architecture Pascal or more recent), you can use mixed precision training with PyTorch 1.6.0 or latest, or by installing the Apex library for previous versions. Just add the flag --fp16 to your command launching one of the scripts mentioned above!
Using mixed precision training usually results in 2x-speedup for training with the same final results (as shown in this table for text classification).
Running on TPUs
When using Tensorflow, TPUs are supported out of the box as a tf.distribute.Strategy.
When using PyTorch, we support TPUs thanks to pytorch/xla. For more context and information on how to setup your TPU environment refer to Google's documentation and to the very detailed pytorch/xla README.
In this repo, we provide a very simple launcher script named xla_spawn.py that lets you run our example scripts on multiple TPU cores without any boilerplate. Just pass a --num_cores flag to this script, then your regular training script with its arguments (this is similar to the torch.distributed.launch helper for torch.distributed):
bash python xla_spawn.py --num_cores num_tpu_you_have \ path_to_script.py \ --all_arguments_of_the_script
As an example, here is how you would fine-tune the BERT large model (with whole word masking) on the text classification MNLI task using the run_glue script, with 8 TPUs:
bash python xla_spawn.py --num_cores 8 \ text-classification/run_glue.py \ --model_name_or_path bert-large-uncased-whole-word-masking \ --task_name mnli \ --do_train \ --do_eval \ --max_seq_length 128 \ --per_device_train_batch_size 8 \ --learning_rate 2e-5 \ --num_train_epochs 3.0 \ --output_dir /tmp/mnli_output/
Logging & Experiment tracking
You can easily log and monitor your runs code. The following are currently supported:
Weights & Biases
To use Weights & Biases, install the wandb package with:
bash pip install wandb
Then log in the command line:
bash wandb login
If you are in Jupyter or Colab, you should login with:
python import wandb wandb.login()
To enable logging to W&B, include "wandb" in the report_to of your TrainingArguments or script. Or just pass along --report_to all if you have wandb installed.
Whenever you use Trainer or TFTrainer classes, your losses, evaluation metrics, model topology and gradients (for Trainer only) will automatically be logged.
Advanced configuration is possible by setting environment variables:
Set run names with run_name argument present in scripts or as part of TrainingArguments.
Additional configuration options are available through generic wandb environment variables.
Refer to related documentation & examples.
Comet.ml
To use comet_ml, install the Python package with:
bash pip install comet_ml
or if in a Conda environment:
bash conda install -c comet_ml -c anaconda -c conda-forge comet_ml