constantinSch/evaluation_summarization

1---2title: Evaluation Summarization3emoji: ⚡4colorFrom: gray5colorTo: yellow6sdk: docker7app_port: 78608pinned: false9short_description: Evaluation Zusammenfassung Rundfunktranskripte10---11 12# Evaluation Summarization Space13 14This repository contains a Docker-based Hugging Face Space for evaluating generated summaries against source transcripts. The app serves a lightweight annotation interface backed by Flask and stores submitted judgements in SQLite.15 16## What the Space does17 18- Presents evaluation items from a JSONL dataset.19- Lets annotators score summary quality across multiple criteria.20- Persists annotations in a SQLite database.21- Exports collected annotations as JSONL.22- Supports optional password protection through a Space secret.23 24## Runtime model25 26This Space uses the `docker` SDK and starts the Flask app defined in [app.py](app.py). The container exposes port `7860`, which is declared in the YAML front matter above and in [Dockerfile](Dockerfile).27 28At runtime, the app reads:29 30- `DATASET_PATH` for the evaluation dataset JSONL31- `DB_PATH` for the SQLite annotations database32- `APP_PASSWORD` for optional login protection33- `SECRET_KEY` for stable HMAC token generation34 35By default, the Docker image is configured for Hugging Face persistent storage mounted at `/data`:36 37- dataset: `/data/2026-04-23_prompt_evaluation_dataset.jsonl`38- annotations DB: `/data/annotations.db`39 40## Deploying on Hugging Face Spaces41 421. Create a new Space using the `Docker` SDK.432. Add persistent storage mounted at `/data`.443. Set the secret `APP_PASSWORD` if the UI should require a login.454. Set the secret `SECRET_KEY` if you want authentication tokens to remain valid across container restarts.465. Push this repository to the Space.476. Upload the dataset JSONL into `/data/` in the Space storage browser.487. Confirm that the uploaded dataset filename matches `DATASET_PATH` in [Dockerfile](Dockerfile).49 50Without persistent storage, annotations stored in SQLite will be lost when the container filesystem is replaced.51 52## Recovering a damaged annotations database53 54If `PRAGMA integrity_check;` reports corruption, stop the app first and work on a copy of the database files, not on the live files in `/data`.55 56Run the recovery script from the repository root:57 58```powershell59uv run python recover_annotations_db.py data/annotations.db data/annotations.recovered.db --force60```61 62The script stages `annotations.db` plus sibling `-wal` and `-shm` files into a temporary directory, copies all readable rows from the `annotations` table into a new clean database, and prints a recovery report including integrity-check results.63 64After a successful run:65 661. stop the app672. keep a backup of the old `annotations.db`, `annotations.db-wal`, and `annotations.db-shm`683. replace only `annotations.db` with the recovered file694. remove old `annotations.db-wal` and `annotations.db-shm`705. restart the app71 72The application now uses a real SQLite upsert for annotation writes, so updating an existing `eval_id` no longer performs the delete-and-reinsert behavior of `INSERT OR REPLACE`.73 74## Local development75 76The app can also run locally. In local development, it defaults to the dataset file in the project root when `DATASET_PATH` is unset.77 78Example environment variables:79 80```powershell81$env:DATASET_PATH = "2026-04-23_prompt_evaluation_dataset.jsonl"82$env:DB_PATH = "annotations.db"83$env:APP_PASSWORD = "your-password"84python app.py85```86 87The Python dependency is defined in [pyproject.toml](pyproject.toml). The container build uses `uv` as configured in [Dockerfile](Dockerfile).88 89## Data expectations90 91The dataset is expected to be a JSONL file with one evaluation item per line. The application relies on stable item IDs and the text fields required to render the transcript, summary, and metadata shown in the UI.92 93Submitted annotations are stored by `eval_id` and include these fields:94 95- `bewertung`96- `korrekt`97- `relevant`98- `vollstaendig`99- `kohaerenz`100- `anmerkungen`101 102If you replace the dataset with a new file that uses different `eval_id` values, existing annotation rows in the SQLite database will no longer line up with the new items.103 104## Notes for duplication105 106If someone duplicates this Space, the most important setup steps are:107 1081. mount persistent storage at `/data`1092. upload a dataset JSONL file to `/data`1103. keep `DATASET_PATH` aligned with the uploaded filename1114. configure `APP_PASSWORD` if access should be restricted112 113For Hugging Face Space metadata options, see the Spaces configuration reference:114 115https://huggingface.co/docs/hub/spaces-config-reference116