CoolFace
Apppublic

constantinSch/evaluation_summarization

sourceHugging Faceupdated 1mo agoView on Hugging Face
0likes
App README

Evaluation Summarization Space

This repository contains a Docker-based Hugging Face Space for evaluating generated summaries against source transcripts. The app serves a lightweight annotation interface backed by Flask and stores submitted judgements in SQLite.

What the Space does

  • —Presents evaluation items from a JSONL dataset.
  • —Lets annotators score summary quality across multiple criteria.
  • —Persists annotations in a SQLite database.
  • —Exports collected annotations as JSONL.
  • —Supports optional password protection through a Space secret.

Runtime model

This Space uses the docker SDK and starts the Flask app defined in app.py. The container exposes port 7860, which is declared in the YAML front matter above and in Dockerfile.

At runtime, the app reads:

  • —DATASET_PATH for the evaluation dataset JSONL
  • —DB_PATH for the SQLite annotations database
  • —APP_PASSWORD for optional login protection
  • —SECRET_KEY for stable HMAC token generation

By default, the Docker image is configured for Hugging Face persistent storage mounted at /data:

  • —dataset: /data/2026-04-23_prompt_evaluation_dataset.jsonl
  • —annotations DB: /data/annotations.db

Deploying on Hugging Face Spaces

  1. 1.Create a new Space using the Docker SDK.
  2. 2.Add persistent storage mounted at /data.
  3. 3.Set the secret APP_PASSWORD if the UI should require a login.
  4. 4.Set the secret SECRET_KEY if you want authentication tokens to remain valid across container restarts.
  5. 5.Push this repository to the Space.
  6. 6.Upload the dataset JSONL into /data/ in the Space storage browser.
  7. 7.Confirm that the uploaded dataset filename matches DATASET_PATH in Dockerfile.

Without persistent storage, annotations stored in SQLite will be lost when the container filesystem is replaced.

Recovering a damaged annotations database

If PRAGMA integrity_check; reports corruption, stop the app first and work on a copy of the database files, not on the live files in /data.

Run the recovery script from the repository root:

powershell
uv run python recover_annotations_db.py data/annotations.db data/annotations.recovered.db --force

The script stages annotations.db plus sibling -wal and -shm files into a temporary directory, copies all readable rows from the annotations table into a new clean database, and prints a recovery report including integrity-check results.

After a successful run:

  1. 1.stop the app
  2. 2.keep a backup of the old annotations.db, annotations.db-wal, and annotations.db-shm
  3. 3.replace only annotations.db with the recovered file
  4. 4.remove old annotations.db-wal and annotations.db-shm
  5. 5.restart the app

The application now uses a real SQLite upsert for annotation writes, so updating an existing eval_id no longer performs the delete-and-reinsert behavior of INSERT OR REPLACE.

Local development

The app can also run locally. In local development, it defaults to the dataset file in the project root when DATASET_PATH is unset.

Example environment variables:

powershell
$env:DATASET_PATH = "2026-04-23_prompt_evaluation_dataset.jsonl"
$env:DB_PATH = "annotations.db"
$env:APP_PASSWORD = "your-password"
python app.py

The Python dependency is defined in pyproject.toml. The container build uses uv as configured in Dockerfile.

Data expectations

The dataset is expected to be a JSONL file with one evaluation item per line. The application relies on stable item IDs and the text fields required to render the transcript, summary, and metadata shown in the UI.

Submitted annotations are stored by eval_id and include these fields:

  • —bewertung
  • —korrekt
  • —relevant
  • —vollstaendig
  • —kohaerenz
  • —anmerkungen

If you replace the dataset with a new file that uses different eval_id values, existing annotation rows in the SQLite database will no longer line up with the new items.

Notes for duplication

If someone duplicates this Space, the most important setup steps are:

  1. 1.mount persistent storage at /data
  2. 2.upload a dataset JSONL file to /data
  3. 3.keep DATASET_PATH aligned with the uploaded filename
  4. 4.configure APP_PASSWORD if access should be restricted

For Hugging Face Space metadata options, see the Spaces configuration reference:

https://huggingface.co/docs/hub/spaces-config-reference