autoevaluate/model-evaluator

1---2title: Model Evaluator3emoji: 📊4colorFrom: red5colorTo: red6sdk: streamlit7sdk_version: 1.10.08app_file: app.py9---10 11# Model Evaluator12 13> Submit evaluation jobs to AutoTrain from the Hugging Face Hub14 15**⚠️ This project has been archived. If you want to evaluate LLMs, checkout [this collection](https://huggingface.co/collections/clefourrier/llm-leaderboards-and-benchmarks-✨-64f99d2e11e92ca5568a7cce) of leaderboards.**16 17## Supported tasks18 19The table below shows which tasks are currently supported for evaluation in the AutoTrain backend:20 21| Task                               | Supported |22|:-----------------------------------|:---------:|23| `binary_classification`            |     ✅     |24| `multi_class_classification`       |     ✅     |25| `multi_label_classification`       |     ❌     |26| `entity_extraction`                |     ✅     |27| `extractive_question_answering`    |     ✅     |28| `translation`                      |     ✅     |29| `summarization`                    |     ✅     |30| `image_binary_classification`      |     ✅     |31| `image_multi_class_classification` |     ✅     |32| `text_zero_shot_evaluation`        |     ✅     |33 34 35## Installation36 37To run the application locally, first clone this repository and install the dependencies as follows:38 39```40pip install -r requirements.txt41```42 43Next, copy the example file of environment variables:44 45```46cp .env.template .env47```48 49and set the `HF_TOKEN` variable with a valid API token from the [`autoevaluator`](https://huggingface.co/autoevaluator) bot user. Finally, spin up the application by running:50 51```52streamlit run app.py53```54 55## Usage56 57Evaluation on the Hub involves two main steps:58 591. Submitting an evaluation job via the UI. This creates an AutoTrain project with `N` models for evaluation. At this stage, the dataset is also processed and prepared for evaluation.602. Triggering the evaluation itself once the dataset is processed.61 62From the user perspective, only step (1) is needed since step (2) is handled by a cron job on GitHub Actions that executes the `run_evaluation_jobs.py` script every 15 minutes.63 64See below for details on manually triggering evaluation jobs.65 66### Triggering an evaluation67 68To evaluate the models in an AutoTrain project, run:69 70```71python run_evaluation_jobs.py72```73 74This will download the [`autoevaluate/evaluation-job-logs`](https://huggingface.co/datasets/autoevaluate/evaluation-job-logs) dataset from the Hub and check which evaluation projects are ready for evaluation (i.e. those whose dataset has been processed).75 76## AutoTrain configuration details77 78Models are evaluated by the [`autoevaluator`](https://huggingface.co/autoevaluator) bot user in AutoTrain, with the payload sent to the `AUTOTRAIN_BACKEND_API` environment variable. Evaluation projects are created and run on either the `prod` or `staging` environments. You can view the status of projects in the AutoTrain UI by navigating to one of the links below (ask internally for access to the staging UI):79 80| AutoTrain environment |                                                AutoTrain UI URL                                                |           `AUTOTRAIN_BACKEND_API`            |81|:---------------------:|:--------------------------------------------------------------------------------------------------------------:|:--------------------------------------------:|82|        `prod`         |         [`https://ui.autotrain.huggingface.co/projects`](https://ui.autotrain.huggingface.co/projects)         |     https://api.autotrain.huggingface.co     |83|       `staging`       | [`https://ui-staging.autotrain.huggingface.co/projects`](https://ui-staging.autotrain.huggingface.co/projects) | https://api-staging.autotrain.huggingface.co |84 85 86The current configuration for evaluation jobs running on [Spaces](https://huggingface.co/spaces/autoevaluate/model-evaluator) is:87 88```89AUTOTRAIN_BACKEND_API=https://api.autotrain.huggingface.co90```91 92To evaluate models with a _local_ instance of AutoTrain, change the environment to:93 94```95AUTOTRAIN_BACKEND_API=http://localhost:800096```97 98### Migrating from staging to production (and vice versa)99 100In general, evaluation jobs should run in AutoTrain's `prod` environment, which is defined by the following environment variable:101 102```103AUTOTRAIN_BACKEND_API=https://api.autotrain.huggingface.co104```105 106However, there are times when it is necessary to run evaluation jobs in AutoTrain's `staging` environment (e.g. because a new evaluation pipeline is being deployed). In these cases the corresponding environement variable is:107 108```109AUTOTRAIN_BACKEND_API=https://api-staging.autotrain.huggingface.co110```111 112To migrate between these two environments, update the `AUTOTRAIN_BACKEND_API` in two places:113 114* In the [repo secrets](https://huggingface.co/spaces/autoevaluate/model-evaluator/settings) associated with the `model-evaluator` Space. This will ensure evaluation projects are created in the desired environment.115* In the [GitHub Actions secrets](https://github.com/huggingface/model-evaluator/settings/secrets/actions) associated with this repo. This will ensure that the correct evaluation jobs are approved and launched via the `run_evaluation_jobs.py` script.116