akshathmangudi/SciCode

1---2dataset_info:3  features:4  - name: benchmark5    dtype: string6  - name: artifact_type7    dtype: string8  - name: problem_id9    dtype: string10  - name: test_id11    dtype: string12  - name: variables13    dtype: string14  splits:15  - name: train16    num_bytes: 188938069617    num_examples: 108218  download_size: 82027783919  dataset_size: 188938069620configs:21- config_name: default22  data_files:23  - split: train24    path: data/train-*25license: mit26task_categories:27- question-answering28language:29- en30tags:31- code32pretty_name: SciCode33size_categories:34- 1K<n<10K35---36# Dataset Card for Dataset Name37 38Official Description (from the authors): 39Since language models (LMs) now outperform average humans on many challenging tasks, 40it has become increasingly difficult to develop challenging, high-quality, and realistic evaluations. 41We address this issue by examining LMs' capabilities to generate code for solving real scientific research problems. 42Incorporating input from scientists and AI researchers in 16 diverse natural science sub-fields, 43including mathematics, physics, chemistry, biology, and materials science, we created a scientist-curated coding benchmark, 44SciCode. The problems in SciCode naturally factorize into multiple subproblems, each involving knowledge recall, reasoning, 45and code synthesis. In total, SciCode contains 338 subproblems decomposed from 80 challenging main problems. 46It offers optional descriptions specifying useful scientific background information and scientist-annotated gold-standard solutions 47and test cases for evaluation. Claude3.5-Sonnet, the best-performing model among those tested, 48can solve only 4.6% of the problems in the most realistic setting. We believe that SciCode demonstrates both contemporary LMs' 49progress towards becoming helpful scientific assistants and sheds light on the development and evaluation of scientific AI in the future.50 51This repository contains the ground truth artifacts that's needed for LightEval benchmarks. 52 53The original SciCode numerical evaluation artifacts are provided in54`raw/raw_ground.h5` for reproducibility and parity with the original55SciCode evaluation pipeline.56 57This dataset uses a single split (`train`) as it represents a complete58set of SciCode numerical evaluation artifacts rather than training data.59 60## Dataset Details61 62### Dataset Sources [optional]63 64<!-- Provide the basic links for the dataset. -->65 66- **Repository:** [https://github.com/scicode-bench/SciCode?tab=readme-ov-file]67- **Paper [optional]:** [https://arxiv.org/abs/2407.13168]68 69## Dataset Card Authors70 71The original authors of SciCode benchmark and Akshath Mangudi for 72providing the ground truth artifact.