chendl/compositional_test
1
1# Testing new Hugging Face Deep Learning Container.2 3This document explains the testing strategy for releasing the new Hugging Face Deep Learning Container. AWS maintains 14 days of currency with framework releases. Besides framework releases, AWS release train is bi-weekly on Monday. Code cutoff date for any changes is the Wednesday before release-Monday. 4 5 6## Test Case 1: Releasing a New Version (Minor/Major) of 🤗 Transformers7 8### Requirements: Test should run on Release Candidate for new `transformers` release to validate the new release is compatible with the DLCs. To run these tests you need credentials for the HF SageMaker AWS Account. You can ask @philschmid or @n1t0 to get access.9 10### Run Tests:11 12Before we can run the tests we need to adjust the `requirements.txt` for PyTorch under `/tests/sagemaker/scripts/pytorch` and for TensorFlow under `/tests/sagemaker/scripts/pytorch`. We adjust the branch to the new RC-tag.13 14```15git+https://github.com/huggingface/transformers.git@v4.5.0.rc0 # install main or adjust ist with vX.X.X for installing version specific-transforms16```17 18After we adjusted the `requirements.txt` we can run Amazon SageMaker tests with: 19 20```bash21AWS_PROFILE=<enter-your-profile> make test-sagemaker22```23These tests take around 10-15 minutes to finish. Preferably make a screenshot of the successfully ran tests.24 25### After Transformers Release:26 27After we have released the Release Candidate we need to create a PR at the [Deep Learning Container Repository](https://github.com/aws/deep-learning-containers).28 29**Creating the update PR:**30 311. Update the two latest `buildspec.yaml` config for [PyTorch](https://github.com/aws/deep-learning-containers/tree/master/huggingface/pytorch) and [TensorFlow](https://github.com/aws/deep-learning-containers/tree/master/huggingface/tensorflow). The two latest `buildspec.yaml` are the `buildspec.yaml` without a version tag and the one with the highest framework version, e.g. `buildspec-1-7-1.yml` and not `buildspec-1-6.yml`. 32 33To update the `buildspec.yaml` we need to adjust either the `transformers_version` or the `datasets_version` or both. Example for upgrading to `transformers 4.5.0` and `datasets 1.6.0`.34```yaml35account_id: &ACCOUNT_ID <set-$ACCOUNT_ID-in-environment>36region: ®ION <set-$REGION-in-environment>37base_framework: &BASE_FRAMEWORK pytorch38framework: &FRAMEWORK !join [ "huggingface_", *BASE_FRAMEWORK]39version: &VERSION 1.6.040short_version: &SHORT_VERSION 1.641 42repository_info:43 training_repository: &TRAINING_REPOSITORY44 image_type: &TRAINING_IMAGE_TYPE training45 root: !join [ "huggingface/", *BASE_FRAMEWORK, "/", *TRAINING_IMAGE_TYPE ]46 repository_name: &REPOSITORY_NAME !join ["pr", "-", "huggingface", "-", *BASE_FRAMEWORK, "-", *TRAINING_IMAGE_TYPE]47 repository: &REPOSITORY !join [ *ACCOUNT_ID, .dkr.ecr., *REGION, .amazonaws.com/,48 *REPOSITORY_NAME ]49 50images:51 BuildHuggingFacePytorchGpuPy37Cu110TrainingDockerImage:52 <<: *TRAINING_REPOSITORY53 build: &HUGGINGFACE_PYTORCH_GPU_TRAINING_PY3 false54 image_size_baseline: &IMAGE_SIZE_BASELINE 1500055 device_type: &DEVICE_TYPE gpu56 python_version: &DOCKER_PYTHON_VERSION py357 tag_python_version: &TAG_PYTHON_VERSION py3658 cuda_version: &CUDA_VERSION cu11059 os_version: &OS_VERSION ubuntu18.0460 transformers_version: &TRANSFORMERS_VERSION 4.5.0 # this was adjusted from 4.4.2 to 4.5.061 datasets_version: &DATASETS_VERSION 1.6.0 # this was adjusted from 1.5.0 to 1.6.062 tag: !join [ *VERSION, '-', 'transformers', *TRANSFORMERS_VERSION, '-', *DEVICE_TYPE, '-', *TAG_PYTHON_VERSION, '-',63 *CUDA_VERSION, '-', *OS_VERSION ]64 docker_file: !join [ docker/, *SHORT_VERSION, /, *DOCKER_PYTHON_VERSION, /, 65 *CUDA_VERSION, /Dockerfile., *DEVICE_TYPE ]66```672. In the PR comment describe what test, we ran and with which package versions. Here you can copy the table from [Current Tests](#current-tests). 68 692. In the PR comment describe what test we ran and with which framework versions. Here you can copy the table from [Current Tests](#current-tests). You can take a look at this [PR](https://github.com/aws/deep-learning-containers/pull/1016), which information are needed. 70## Test Case 2: Releasing a New AWS Framework DLC71 72 73## Execute Tests74 75### Requirements:76AWS is going to release new DLCs for PyTorch and/or TensorFlow. The Tests should run on the new framework versions with current `transformers` release to validate the new framework release is compatible with the `transformers` version. To run these tests you need credentials for the HF SageMaker AWS Account. You can ask @philschmid or @n1t0 to get access. AWS will notify us with a new issue in the repository pointing to their framework upgrade PR.77 78### Run Tests:79 80Before we can run the tests we need to adjust the `requirements.txt` for Pytorch under `/tests/sagemaker/scripts/pytorch` and for Tensorflow under `/tests/sagemaker/scripts/pytorch`. We add the new framework version to it.81 82```83torch==1.8.1 # for pytorch84tensorflow-gpu==2.5.0 # for tensorflow85```86 87After we adjusted the `requirements.txt` we can run Amazon SageMaker tests with. 88 89```bash90AWS_PROFILE=<enter-your-profile> make test-sagemaker91```92These tests take around 10-15 minutes to finish. Preferably make a screenshot of the successfully ran tests.93 94### After successful Tests:95 96After we have successfully run tests for the new framework version we need to create a PR at the [Deep Learning Container Repository](https://github.com/aws/deep-learning-containers).97 98**Creating the update PR:**99 1001. Create a new `buildspec.yaml` config for [PyTorch](https://github.com/aws/deep-learning-containers/tree/master/huggingface/pytorch) and [TensorFlow](https://github.com/aws/deep-learning-containers/tree/master/huggingface/tensorflow) and rename the old `buildspec.yaml` to `buildespec-x.x.x`, where `x.x.x` is the base framework version, e.g. if pytorch 1.6.0 is the latest version in `buildspec.yaml` the file should be renamed to `buildspec-yaml-1-6.yaml`. 101 102To create the new `buildspec.yaml` we need to adjust the `version` and the `short_version`. Example for upgrading to `pytorch 1.7.1`. 103 104```yaml105account_id: &ACCOUNT_ID <set-$ACCOUNT_ID-in-environment>106region: ®ION <set-$REGION-in-environment>107base_framework: &BASE_FRAMEWORK pytorch108framework: &FRAMEWORK !join [ "huggingface_", *BASE_FRAMEWORK]109version: &VERSION 1.7.1 # this was adjusted from 1.6.0 to 1.7.1110short_version: &SHORT_VERSION 1.7 # this was adjusted from 1.6 to 1.7111 112repository_info:113 training_repository: &TRAINING_REPOSITORY114 image_type: &TRAINING_IMAGE_TYPE training115 root: !join [ "huggingface/", *BASE_FRAMEWORK, "/", *TRAINING_IMAGE_TYPE ]116 repository_name: &REPOSITORY_NAME !join ["pr", "-", "huggingface", "-", *BASE_FRAMEWORK, "-", *TRAINING_IMAGE_TYPE]117 repository: &REPOSITORY !join [ *ACCOUNT_ID, .dkr.ecr., *REGION, .amazonaws.com/,118 *REPOSITORY_NAME ]119 120images:121 BuildHuggingFacePytorchGpuPy37Cu110TrainingDockerImage:122 <<: *TRAINING_REPOSITORY123 build: &HUGGINGFACE_PYTORCH_GPU_TRAINING_PY3 false124 image_size_baseline: &IMAGE_SIZE_BASELINE 15000125 device_type: &DEVICE_TYPE gpu126 python_version: &DOCKER_PYTHON_VERSION py3127 tag_python_version: &TAG_PYTHON_VERSION py36128 cuda_version: &CUDA_VERSION cu110129 os_version: &OS_VERSION ubuntu18.04130 transformers_version: &TRANSFORMERS_VERSION 4.4.2131 datasets_version: &DATASETS_VERSION 1.5.0132 tag: !join [ *VERSION, '-', 'transformers', *TRANSFORMERS_VERSION, '-', *DEVICE_TYPE, '-', *TAG_PYTHON_VERSION, '-',133 *CUDA_VERSION, '-', *OS_VERSION ]134 docker_file: !join [ docker/, *SHORT_VERSION, /, *DOCKER_PYTHON_VERSION, /, 135 *CUDA_VERSION, /Dockerfile., *DEVICE_TYPE ]136```1372. In the PR comment describe what test we ran and with which framework versions. Here you can copy the table from [Current Tests](#current-tests). You can take a look at this [PR](https://github.com/aws/deep-learning-containers/pull/1025), which information are needed.138 139## Current Tests140 141| ID | Description | Platform | #GPUS | Collected & evaluated metrics |142|-------------------------------------|-------------------------------------------------------------------|-----------------------------|-------|------------------------------------------|143| pytorch-transfromers-test-single | test bert finetuning using BERT fromtransformerlib+PT | SageMaker createTrainingJob | 1 | train_runtime, eval_accuracy & eval_loss |144| pytorch-transfromers-test-2-ddp | test bert finetuning using BERT from transformer lib+ PT DPP | SageMaker createTrainingJob | 16 | train_runtime, eval_accuracy & eval_loss |145| pytorch-transfromers-test-2-smd | test bert finetuning using BERT from transformer lib+ PT SM DDP | SageMaker createTrainingJob | 16 | train_runtime, eval_accuracy & eval_loss |146| pytorch-transfromers-test-1-smp | test roberta finetuning using BERT from transformer lib+ PT SM MP | SageMaker createTrainingJob | 8 | train_runtime, eval_accuracy & eval_loss |147| tensorflow-transfromers-test-single | Test bert finetuning using BERT from transformer lib+TF | SageMaker createTrainingJob | 1 | train_runtime, eval_accuracy & eval_loss |148| tensorflow-transfromers-test-2-smd | test bert finetuning using BERT from transformer lib+ TF SM DDP | SageMaker createTrainingJob | 16 | train_runtime, eval_accuracy & eval_loss |149