kostis-init/CP-Bench-Leaderboard

[!IMPORTANT] CP-Bench has been superseded by DCP-Bench-Open. This repository is kept for archival and reproducibility purposes only. Please use the current version here: DCP-Bench/DCP-Bench-Open.

🚀 CP-Bench Leaderboard

This repository contains the leaderboard of the CP-Bench dataset.

📁 Structure

• —app.py — Launches the Gradio interface.
• —src/ — Contains the main logic for fetching and displaying leaderboard data.'
• —config.py — Configuration for the leaderboard.
• —eval.py — Evaluation logic for model submissions.
• —hf_utils.py — Utilities file.
• —ui.py — UI components for displaying the leaderboard.
• —user_eval.py — The logic for the evaluation of submitted models, it can also be used to evaluate models locally.
• —README.md — (you are here)

🧠 How It Works

1. 1.Users submit a .jsonl file with their generated models
2. 2.The submission is uploaded to a storage repository (Hugging Face Hub).
3. 3.An evaluation script is triggered, which:
4. 4.Loads the submission.
5. 5.Evaluates the models against the benchmark dataset.
6. 6.Computes metrics.
7. 7.The results are stored and displayed on the leaderboard.

🛠️ Development

To run locally:

pip install -r requirements.txt
python app.py

If you wish to contribute or modify the leaderboard, feel free to open discussions or pull requests. For adding more modelling frameworks, please modify the src/user_eval.py file to include the execution code for the new framework.