integer8/GSBench

GSBench

GSBench is a benchmark for evaluating whether AI agents can complete executable genomic selection and genomics-related data analysis tasks from natural-language instructions.

Benchmark content

GSBench contains 46 tasks covering practical genomics and genomic selection scenarios, including genotype quality control, population structure analysis, genetic parameter estimation, genomic prediction, gene mining, functional characterization, and environmental or phenotypic data analysis.

Data source

The data used in GSBench are based on datasets reported in Liu et al. (2020), Liu et al. (2021), and Yang et al. (2019). During benchmark construction, we selected task-relevant data subsets, standardized file formats, and performed the necessary preprocessing according to the requirements of different evaluation tasks.

References:

• —Liu, H.J., Wang, X., Xiao, Y., Luo, J., Qiao, F., Yang, W., Zhang, R., Meng, Y., Sun, J., Yan, S., Peng, Y., Niu, L., Jian, L., Song, W., Yan, J., Li, C., Zhao, Y., Liu, Y., Warburton, M.L., Zhao, J. and Yan, J. (2020). CUBIC: an atlas of genetic architecture promises directed maize improvement. Genome Biology, 21, 20.
• —Liu, N., Du, Y., Warburton, M.L., Xiao, Y., Yan, J. and Michael, P. (2021). Phenotypic plasticity contributes to maize adaptation and heterosis. Molecular Biology and Evolution, 38, 1262-1275.
• —Yang, N., Liu, J., Gao, Q., Gui, S., Chen, L., Yang, L., Huang, J., Deng, T., Luo, J., He, L., Wang, Y., Xu, P., Peng, Y., Shi, Z., Lan, L., Ma, Z., Yang, X., Zhang, Q., Bai, M., Li, S., Li, W., Liu, L., Jackson, D. and Yan, J. (2019). Genome assembly of a tropical maize inbred line provides insights into structural variation and crop improvement. Nature Genetics, 51, 1052-1059.

Repository structure

GSBench/
├── README.md
├── data/
│   ├── GSBench.jsonl
│   └── train-00000-of-00001.parquet
└── files/
│   ├── Q001/
│   ├── Q002/
│   ├── ...
│   └── Q046/

Task file

The canonical task file is:

data/GSBench.jsonl

Each line is one benchmark task. Example:

{
  "id": 1,
  "task": "Given a VCF-formatted genotype file located at {Q001/raw.vcf.gz}, perform original data quality control (QC). Filter out variants with a missing rate >10% and a MAF <0.05. Convert the QC-filtered data into PLINK binary format (geno_qc.bed/bim/fam). Calculate the first five principal components (PCs) and save the proportion of variance explained by each PC to pca_results.txt.",
  "reference_steps": 4,
  "categories": [
    "Data quality control",
    "Population genetic structure analysis"
  ],
  "data_files": [
    "files/Q001/raw.vcf.gz"
  ]
}

Fields

Input files

Input files are stored under:

files/Qxxx/

For example, input files for Q001 are stored in:

files/Q001/

The paths listed in data_files are the files that an AI agent should use to execute the corresponding task.

Dataset statistics

• —Number of tasks: 46

Category counts:

A task may belong to multi categories.