MMSciCode/MMSciCode

MMSciCode

<p align="center"> <a href="#about">About</a> &nbsp;|&nbsp; <a href="#benchmark-construction">Benchmark Construction</a> &nbsp;|&nbsp; <a href="#statistics">Statistics</a> &nbsp;|&nbsp; <a href="#usage">Usage</a> &nbsp;|&nbsp; <a href="#citation">Citation</a> </p>

About

This repository contains MMSciCode, a benchmark for paper-grounded scientific research coding. MMSciCode evaluates whether a model can recover masked core functions from real research code using the surrounding repository context, paper-derived context, and sample-specific implementation metadata.

The benchmark spans Python, R, and C/C++ projects collected from scientific papers and their associated code releases. Each task is evaluated by inserting the generated function back into the original project and running the corresponding unit tests.

This Hugging Face dataset repository contains both the benchmark data and the Dockerfile build assets used to reproduce the execution environments.

Benchmark Construction

MMSciCode is built from real scientific software projects through a function-level construction pipeline:

1. 1.Paper and code collection: scientific papers are paired with their released code repositories.
2. 2.Function extraction: candidate functions are extracted from each project together with file paths, line numbers, and repository structure metadata.
3. 3.Core-function selection: expert annotations identify functions that implement paper-relevant algorithms, equations, simulations, or analysis procedures.
4. 4.Task creation: selected functions are masked while preserving the surrounding code context and paper-derived implementation evidence.
5. 5.Executable validation: generated implementations are inserted back into the original project and checked with sample-specific tests.
6. 6.Environment packaging: Dockerfile build contexts are provided for the exported execution environments.

Why MMSciCode?

Scientific coding differs from short standalone programming tasks: solutions must often match paper-specific notation, domain assumptions, project-local APIs, numerical behavior, and existing repository structure. MMSciCode is designed to measure those abilities directly by using real research code and containerized execution.

Each sample directory under a language-level data/ folder includes metadata such as available functions, selected core functions, paper or article context, repository structure, and test status.

Statistics

Dockerfile environment directories are organized by language-level dockerfiles/ folders:

Repository Layout

MMSciCode/
  Python/
    data/
      <sample_id>/
    dockerfiles/
      <environment_id>/
  R/
    data/
      <sample_id>/
    dockerfiles/
      <environment_id>/
  C_CPP/
    data/
      <sample_id>/
    dockerfiles/
      <environment_id>/
  manifest.jsonl
  index.tsv
  build_all_serial.sh
  distributable_env_dockerfiles.tar.gz

Each sample directory contains the following files. Required files are present in every sample; optional files are present when applicable.

Field-level notes for unit_test_status.json:

• —target_functions[].line_start / line_end are optional and may be null; they are informational and are not consumed by the evaluation pipeline (function location is resolved from selected_core_functions.json).
• —target_functions[].test_file is optional and may be empty for samples whose harness discovers tests by convention.
• —legacy_backup and validation hold historical build/validation records and are not required to run the benchmark.

The root files:

• —manifest.jsonl — one row per benchmark task (624 rows) with language, sample_id, func_index, paper_id, paper_title, subject, function_name, file_path, paper_section, conda_env, and docker_image. A task is uniquely identified by (sample_id, func_index), where func_index is the 0-based position in that sample's selected_core_functions.json. file_path is the sample-relative path of the file the function lives in and is tested in. This is also the file rendered by the Dataset Viewer.
• —index.tsv — maps each Docker image name to its environment id and Dockerfile directory (image, env, dir, editable). The editable flag is Docker build metadata (whether the environment installs the project as an editable package); it does not indicate whether a task may be modified.
• —distributable_env_dockerfiles.tar.gz — the same Dockerfile assets as a standalone package.

Note on environments (Docker and conda). Docker does not replace conda here — each Docker image is a named conda environment packaged in a container. Every image is built by installing micromamba and creating the environment named by conda_env from a pinned dependency lockfile (see the dockerfiles/<environment_id>/Dockerfile, built with --build-arg ENV_NAME=<conda_env>). The two manifest columns are therefore complementary, not redundant: - conda_env — the authoritative environment name (mirrors unit_test_status.json → environment.conda_env_name) that the test runner activates before running a sample's tests. - docker_image — the prebuilt container that ships that environment (a 1:1 mapping via index.tsv). Both reproduction paths use conda_env: run docker_image (the environment is already created inside it and gets activated), or recreate conda_env locally from the Dockerfile's package spec. The runner activates the named conda environment either way. Note on original project code. Each sample bundles a real research project. That upstream source may contain the original authors' absolute paths, machine-specific comments, or non-English text. These are part of the preserved research artifact and are intentionally left unmodified; the MMSciCode-generated metadata files above have been scrubbed of any build-time paths. Note on standalone C/C++ samples. A small number of C/C++ samples compile with the system toolchain and have no conda_env_name (and therefore no Docker image / empty docker_image in the manifest). They are evaluated with a local gcc/g++ + cmake build rather than a prebuilt container.

Usage

Downloading the Dataset

git lfs install
git clone https://huggingface.co/datasets/MMSciCode/MMSciCode
cd MMSciCode

Or download with huggingface_hub:

from huggingface_hub import snapshot_download

dataset_dir = snapshot_download(
    repo_id="MMSciCode/MMSciCode",
    repo_type="dataset",
)

Inspecting a Task

Each benchmark task is defined inside a language-specific data/ directory. For example:

ls Python/data/<sample_id>
cat Python/data/<sample_id>/selected_core_functions.json
cat Python/data/<sample_id>/unit_test_status.json

selected_core_functions.json describes the functions selected for evaluation, including their source locations, natural-language descriptions, paper references, and implementation cues.

Building Docker Environments

The repository includes Dockerfile build contexts under the language-level dockerfiles/ directories. To build all indexed environments serially:

chmod +x build_all_serial.sh
./build_all_serial.sh

The build script reads index.tsv, locates each Dockerfile directory under Python/dockerfiles/, R/dockerfiles/, or C_CPP/dockerfiles/, and tags the resulting image with the name listed in the index.

Optional build arguments can be passed through environment variables:

CONDA_MIRROR=https://mirrors.tuna.tsinghua.edu.cn/anaconda ./build_all_serial.sh
PIP_STRICT=1 ./build_all_serial.sh

Using the Standalone Dockerfile Package

If you only need the Dockerfile build contexts, extract the bundled archive:

tar -xzf distributable_env_dockerfiles.tar.gz
cd distributable_env_dockerfiles
./build_all_serial.sh

Links

Citation

If you find MMSciCode useful in your research, please cite our paper:

@inproceedings{xia-etal-2026-mmscicode,
    title = "{MMS}ci{C}ode: Real-world Evaluation of Multilingual Multi-Discipline Scientific Research Coding",
    author = "Xia, Xue and Yang, Zheyuan and Cohan, Arman and Zhao, Yilun",
    editor = "Liakata, Maria and Moreira, Viviane P. and Zhang, Jiajun and Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.acl-long.1566/",
    doi = "10.18653/v1/2026.acl-long.1566",
    pages = "33981--33999",
    ISBN = "979-8-89176-390-6"
}