akshathmangudi/SciCode
Dataset Card for Dataset Name Official Description (from the authors): Since language models (LMs) now outperform average humans on many challenging tasks, it has become increasingly difficult to develop challenging, high-quality, and realistic evaluations. We address this issue by examining LMs' capabilities to generate code for solving real scientific research problems. Incorporating input from scientists and AI researchers in 16 diverse natural science sub-fields… See the full description on the dataset page: https://huggingface.co/datasets/akshathmangudi/SciCode.
Dataset Card for Dataset Name
Official Description (from the authors): Since language models (LMs) now outperform average humans on many challenging tasks, it has become increasingly difficult to develop challenging, high-quality, and realistic evaluations. We address this issue by examining LMs' capabilities to generate code for solving real scientific research problems. Incorporating input from scientists and AI researchers in 16 diverse natural science sub-fields, including mathematics, physics, chemistry, biology, and materials science, we created a scientist-curated coding benchmark, SciCode. The problems in SciCode naturally factorize into multiple subproblems, each involving knowledge recall, reasoning, and code synthesis. In total, SciCode contains 338 subproblems decomposed from 80 challenging main problems. It offers optional descriptions specifying useful scientific background information and scientist-annotated gold-standard solutions and test cases for evaluation. Claude3.5-Sonnet, the best-performing model among those tested, can solve only 4.6% of the problems in the most realistic setting. We believe that SciCode demonstrates both contemporary LMs' progress towards becoming helpful scientific assistants and sheds light on the development and evaluation of scientific AI in the future.
This repository contains the ground truth artifacts that's needed for LightEval benchmarks.
The original SciCode numerical evaluation artifacts are provided in raw/raw_ground.h5 for reproducibility and parity with the original SciCode evaluation pipeline.
This dataset uses a single split (train) as it represents a complete set of SciCode numerical evaluation artifacts rather than training data.
Dataset Details
Dataset Sources [optional]
<!-- Provide the basic links for the dataset. -->
- Repository: [https://github.com/scicode-bench/SciCode?tab=readme-ov-file]
- Paper [optional]: [https://arxiv.org/abs/2407.13168]
Dataset Card Authors
The original authors of SciCode benchmark and Akshath Mangudi for providing the ground truth artifact.