HadrienByr/model-evaluator

1---2title: Model Evaluator3emoji: 📊4colorFrom: red5colorTo: red6sdk: streamlit7sdk_version: 1.10.08app_file: app.py9duplicated_from: autoevaluate/model-evaluator10---11 12# Model Evaluator13 14> Submit evaluation jobs to AutoTrain from the Hugging Face Hub15 16## Supported tasks17 18The table below shows which tasks are currently supported for evaluation in the AutoTrain backend:19 20| Task                               | Supported |21|:-----------------------------------|:---------:|22| `binary_classification`            |     ✅     |23| `multi_class_classification`       |     ✅     |24| `multi_label_classification`       |     ❌     |25| `entity_extraction`                |     ✅     |26| `extractive_question_answering`    |     ✅     |27| `translation`                      |     ✅     |28| `summarization`                    |     ✅     |29| `image_binary_classification`      |     ✅     |30| `image_multi_class_classification` |     ✅     |31| `text_zero_shot_evaluation`        |     ✅     |32 33 34## Installation35 36To run the application locally, first clone this repository and install the dependencies as follows:37 38```39pip install -r requirements.txt40```41 42Next, copy the example file of environment variables:43 44```45cp .env.template .env46```47 48and set the `HF_TOKEN` variable with a valid API token from the [`autoevaluator`](https://huggingface.co/autoevaluator) bot user. Finally, spin up the application by running:49 50```51streamlit run app.py52```53 54## Usage55 56Evaluation on the Hub involves two main steps:57 581. Submitting an evaluation job via the UI. This creates an AutoTrain project with `N` models for evaluation. At this stage, the dataset is also processed and prepared for evaluation.592. Triggering the evaluation itself once the dataset is processed.60 61From the user perspective, only step (1) is needed since step (2) is handled by a cron job on GitHub Actions that executes the `run_evaluation_jobs.py` script every 15 minutes.62 63See below for details on manually triggering evaluation jobs.64 65### Triggering an evaluation66 67To evaluate the models in an AutoTrain project, run:68 69```70python run_evaluation_jobs.py71```72 73This will download the [`autoevaluate/evaluation-job-logs`](https://huggingface.co/datasets/autoevaluate/evaluation-job-logs) dataset from the Hub and check which evaluation projects are ready for evaluation (i.e. those whose dataset has been processed).74 75## AutoTrain configuration details76 77Models are evaluated by the [`autoevaluator`](https://huggingface.co/autoevaluator) bot user in AutoTrain, with the payload sent to the `AUTOTRAIN_BACKEND_API` environment variable. Evaluation projects are created and run on either the `prod` or `staging` environments. You can view the status of projects in the AutoTrain UI by navigating to one of the links below (ask internally for access to the staging UI):78 79| AutoTrain environment |                                                AutoTrain UI URL                                                |           `AUTOTRAIN_BACKEND_API`            |80|:---------------------:|:--------------------------------------------------------------------------------------------------------------:|:--------------------------------------------:|81|        `prod`         |         [`https://ui.autotrain.huggingface.co/projects`](https://ui.autotrain.huggingface.co/projects)         |     https://api.autotrain.huggingface.co     |82|       `staging`       | [`https://ui-staging.autotrain.huggingface.co/projects`](https://ui-staging.autotrain.huggingface.co/projects) | https://api-staging.autotrain.huggingface.co |83 84 85The current configuration for evaluation jobs running on [Spaces](https://huggingface.co/spaces/autoevaluate/model-evaluator) is:86 87```88AUTOTRAIN_BACKEND_API=https://api.autotrain.huggingface.co89```90 91To evaluate models with a _local_ instance of AutoTrain, change the environment to:92 93```94AUTOTRAIN_BACKEND_API=http://localhost:800095```96 97### Migrating from staging to production (and vice versa)98 99In general, evaluation jobs should run in AutoTrain's `prod` environment, which is defined by the following environment variable:100 101```102AUTOTRAIN_BACKEND_API=https://api.autotrain.huggingface.co103```104 105However, there are times when it is necessary to run evaluation jobs in AutoTrain's `staging` environment (e.g. because a new evaluation pipeline is being deployed). In these cases the corresponding environement variable is:106 107```108AUTOTRAIN_BACKEND_API=https://api-staging.autotrain.huggingface.co109```110 111To migrate between these two environments, update the `AUTOTRAIN_BACKEND_API` in two places:112 113* In the [repo secrets](https://huggingface.co/spaces/autoevaluate/model-evaluator/settings) associated with the `model-evaluator` Space. This will ensure evaluation projects are created in the desired environment.114* In the [GitHub Actions secrets](https://github.com/huggingface/model-evaluator/settings/secrets/actions) associated with this repo. This will ensure that the correct evaluation jobs are approved and launched via the `run_evaluation_jobs.py` script.115