apodex/FrontierChallenge

1---2license: cc-by-4.03language:4- en5task_categories:6- other7tags:8- scientific-agents9- benchmark10- evaluation11pretty_name: FrontierChallenge12configs:13- config_name: default14  data_files:15  - split: train16    path: manifest.jsonl17---18 19# FrontierChallenge20 21FrontierChallenge provides **97 scientific workflow tasks** with plaintext22English instructions, inputs, Harbor definitions, domain labels, and the23redistributable open runtime image.24 25| Path | Contents |26|---|---|27| `manifest.jsonl` | Dataset Viewer rows with task ID, taxonomy, difficulty, runtime, and instruction |28| `tasks/<task-id>/` | `instruction.md`, task metadata, environment definition, and agent-visible inputs |29| `images/` | Verified `linux/amd64` Docker archive for the 81 open-image tasks |30 31This repository contains no graders, rubrics, fixtures, or reference outputs.32Those are stored as encrypted archives in the separate33[`apodex/FrontierChallenge-reference`](https://huggingface.co/datasets/apodex/FrontierChallenge-reference)34dataset.35 36Use the [FrontierChallenge runtime](https://github.com/ApodexAI/FrontierAgent/tree/main/benchmarks/frontierchallenge)37to download both datasets, verify their shared registry, load the image archive,38run Harbor, and score a task. The evaluator host needs Python 3.12+ for the39pinned Harbor 0.20.0; task containers keep their own frozen Python versions.40 41```bash42git clone https://github.com/ApodexAI/FrontierAgent.git43cd FrontierAgent/benchmarks/frontierchallenge44python3.12 -m venv .venv45source .venv/bin/activate46python -m pip install -e .47HF_TOKEN=hf_... ./scripts/setup.sh --track open48cp .env.example .env49# Fill model and judge credentials in .env before running.50./scripts/run_eval.sh \51  --agent claude-code --model <model> \52  --include-task-name task_011_cell_migration_wound_healing53```54 55The 16 task definitions that execute ORCA and their inputs are included56normally. ORCA itself and an ORCA-configured image are never distributed;57evaluators obtain ORCA from its official provider and follow the runtime58repository's [local-image tutorial](https://github.com/ApodexAI/FrontierAgent/blob/main/benchmarks/frontierchallenge/docs/providers/orca.md).59 60Track membership follows what a task executes, not software named in supplied61files. For example, `task_098_orca_claisen_thermochemistry` reads precomputed62ORCA output without running ORCA, so it remains in the open track.63 64Verify a downloaded solve package with:65 66```bash67python tools/verify_dataset.py68```69 70## Metrics71 72Official Pass Rate uses a **strict score threshold**: a task passes only when73`evaluation_complete == 1` and valid `task_score > 0.999`. Native per-task74thresholds do not determine this metric.75Use unrounded scores: exactly `0.999` does not pass; `0.9991` and `1.0` pass.76 77- **Pass Rate (%)** = 100 × number of completed tasks with valid `task_score > 0.999` / 97.78- **Score (0–100)** = 100 × sum of valid, completed `task_score` values / 97.79- Partial credit remains the native rubric score normalized to [0, 1].80  Missing tasks, invalid scores, and incomplete evaluations contribute zero81  to the fixed denominator and must be reported separately; incomplete runs82  are not final benchmark results.83- Use one predeclared attempt per task. A subset may use its predeclared task84  count as denominator, but must be labeled as a subset, not the 97-task result.85 86There is only one pass field, `passed`, using the same strict threshold in87`reward.json`, `summary.csv`, and `summary.json`. No alternate pass field is88emitted. The runtime applies this rule to the staged reward adapter after89unsealing; encrypted archives and partial-credit rubrics remain unchanged.90The updated runtime also removes native pass decisions from published verifier91diagnostics. Jobs created by older runtimes cannot be resumed under the new92policy: use a fresh job name. Old files are left untouched; summarizing an old93job recomputes the official metric but does not rewrite its logs or rewards.94The summary records `metric_definition: score-gt-0.999`. Historical results95using per-task thresholds, `== 1.0`, or `>= 0.999` must be recomputed from raw96scores before comparison. See the97[scoring guide](https://github.com/ApodexAI/FrontierAgent/blob/main/benchmarks/frontierchallenge/docs/scoring.md)98and [runtime update](https://github.com/ApodexAI/FrontierAgent/pull/30).99 100**Runtime rollout:** the runtime update is pending. Before reporting results,101check that your summary records `metric_definition: score-gt-0.999`;102older runtimes use the legacy rule. Use the runtime summary for the fixed10397-task denominator; Harbor may aggregate only attempted trials.104 105## Integrity106 107The root `README.md` is a mutable dataset card and is intentionally outside108`checksums.sha256`. Task files (including task-level READMEs), manifests,109registries, and verification tools remain checksummed. Payload changes require110regenerating their checksum entries; editing this card does not.111 112## Citation113 114```bibtex115@misc{apodex11,116  title         = {Apodex 1.1: Scaling Agentic Intelligence for Complex Work},117  author        = {{Apodex Team}},118  year          = {2026},119  eprint        = {2608.23283},120  archivePrefix = {arXiv},121  primaryClass  = {cs.AI},122  url           = {https://arxiv.org/abs/2608.23283}123}124 125@misc{frontierchallenge,126  title         = {FrontierChallenge: Evaluating Scientific Workflow Completion},127  author        = {Liangcai Su and Zhaopeng Feng and Zhuo Chen and Zhen Zhang128                   and Xiang Lin and Ruilin Li and Handuo Zhang and Ning Wang129                   and Kailong Wen and Yueqi Guo and Feng Xing and Yiling Guo130                   and Chenxiong Qian and Simon Shaolei Du and Lidong Bing131                   and Xinyu Wang},132  year          = {2026},133  eprint        = {2608.24979},134  archivePrefix = {arXiv},135  primaryClass  = {cs.AI},136  url           = {https://arxiv.org/abs/2608.24979}137}138```139