cabbage972/GitChameleon-2.0

GitChameleon 2.0

GitChameleon 2.0 is an AI coding benchmark comprising 328 Python-based problems conditioned on specific versions of popular libraries for scientific computing and web development. It evaluates whether AI code generation models can correctly use library APIs as they existed at a particular version — a challenging test of version-specific knowledge.

Note: This is GitChameleon 2.0, a distinct and newer work from the original GitChameleon benchmark. Please do not confuse the two.

Project website: gitchameleon-2-0.github.io — paper, results, citation, and getting started guide.

Paper: GitChameleon 2.0: Evaluating AI Code Generation Against Python Library Version Incompatibilities (ACL 2026, Main)

Example Task

Each problem provides a library version, a natural-language description, and a stub to complete:

Library: torch==1.9.0 | Python: 3.7 Problem: Calculate the logarithm of the cumulative distribution function of the standard normal distribution using available functions. If not available in PyTorch, use another library.

import torch
def log_ndtr(input_tensor: torch.Tensor) -> torch.Tensor:
    # your solution here

The model must produce a solution that passes the visible test:

from scipy.stats import norm
input_tensor = torch.linspace(-10, 10, steps=20)
expected_result = torch.tensor([-5.3231e+01, ..., -7.6199e-24], dtype=torch.float64)
assert torch.allclose(log_ndtr(input_tensor), expected_result, rtol=1e-3, atol=1e-3)

This particular problem tests awareness that torch.special.log_ndtr was not available in torch==1.9.0, requiring the model to fall back to scipy.stats.norm.logcdf.

Dataset Configs

Usage

from datasets import load_dataset

# Load problems
ds = load_dataset("cabbage972/GitChameleon-2.0", "problems")

# Load ground-truth solutions
solutions = load_dataset("cabbage972/GitChameleon-2.0", "solutions")

Schema

problems config

solutions config

Running Evaluation

Evaluation is run via the GitChameleonBenchmark harness. Requirements: Python 3.9+, Poetry, and Docker.

git clone https://github.com/mrcabbage972/GitChameleonBenchmark.git
cd GitChameleonBenchmark
make evals-setup
evaluate --solution-path SOLUTION_PATH [--workers WORKERS]

Your solution file should be a JSONL where each line has example_id and answer fields (matching the solutions config schema above). Success rates are printed to stdout and detailed logs are written next to the solution file.

Libraries Covered

26 libraries including: torch, scipy, sympy, flask, falcon, numpy, scikit-learn, pandas, django, librosa, and more.

Citation

@misc{misra2025gitchameleon20evaluatingai,
      title={GitChameleon 2.0: Evaluating AI Code Generation Against Python Library Version Incompatibilities},
      author={Diganta Misra and Nizar Islah and Victor May and Brice Rauby and Zihan Wang and Justine Gehring and Antonio Orvieto and Muawiz Chaudhary and Eilif B. Muller and Irina Rish and Samira Ebrahimi Kahou and Massimo Caccia},
      year={2025},
      eprint={2507.12367},
      archivePrefix={arXiv},
      primaryClass={cs.SE},
      url={https://arxiv.org/abs/2507.12367},
}