mercury
Datasets
All datasets matching “mercury”Mercury_MultilingualMercury
Welcome to Mercury 🪐!
This is the dataset of the paper 📃 Mercury: A Code Efficiency Benchmark for Code Large Language Models
Mercury is the first code efficiency benchmark designed for code synthesis tasks.
It consists of 1,889 programming tasks covering diverse difficulty levels, along with test case generators that produce unlimited cases for comprehensive evaluation.
How to use Mercury Evaluation
git clone https://github.com/Elfsong/Mercury_Eval.git
cd… See the full description on the dataset page: https://huggingface.co/datasets/Elfsong/Mercury.PyInstruct PyBench: Evaluate LLM Agent on Real World Tasks
📃 Paper
•
🤗 Data (PyInstruct)
•
🤗 Model (PyLlama3)
•
Code
•
PyBench is a comprehensive benchmark evaluating LLM on real-world coding tasks including chart analysis, text analysis, image/ audio editing, complex math and software/website development. We collect files from Kaggle, arXiv, and other sources and automatically generate queries according to the type and content of each file.
Why PyBench?
The LLM Agent, equipped… See the full description on the dataset page: https://huggingface.co/datasets/Mercury7353/PyInstruct.osworld-tutorial-ragmercury_verl
Mercury (verl efficiency eval set)
The eval split of Elfsong/Mercury
(arXiv 2402.07844; 256 LeetCode-style tasks;
the train split ships no test cases and is not gradable), converted to the verl
rule-reward schema by verl/scripts/data/mercury.py. Source license
CC-BY-NC-4.0 (non-commercial) -- this conversion keeps that license.
Every row's ground truth carries the full official scoring contract: entry
point, the task's convert_offline/evaluate_offline hooks (lctk linked-list /… See the full description on the dataset page: https://huggingface.co/datasets/OctoReasoner/mercury_verl.store_block