MiniMaxAI/OctoCodingBench

OctoCodingBench: Instruction-Following Benchmark for Coding Agents

English | 中文

🌟 Overview

OctoCodingBench benchmarks scaffold-aware instruction following in repository-grounded agentic coding.

Why OctoCodingBench?

Existing benchmarks (SWE-bench, etc.) focus on task completion — whether the agent produces correct code. However, they miss a critical dimension: does the agent follow the rules while solving the task?

In real-world agentic coding, agents must comply with:

• —System-level behavioral constraints (e.g., no emoji, specific output formats)
• —Project coding conventions (CLAUDE.md, AGENTS.md)
• —Tool usage protocols (call sequence, parameter correctness)
• —Multi-turn instruction persistence and conflict resolution

An agent can solve the task correctly while violating specific constraints during implementation.

Instruction Sources

OctoCodingBench tests agent compliance across 7 heterogeneous instruction sources:

🚀 Key Features

• —Disentangle Task Completion from Rule Following: High task success ≠ high instruction compliance
• —Multi-Source Heterogeneous Constraints: 7 distinct instruction categories with different authority levels
• —Binary Checklist Scoring: Each check is objectively decidable (pass/fail)
• —Multi-Scaffold Support: Claude Code, Kilo, Droid — real production scaffolds
• —Conflict Detection: Tests how agents resolve contradictory instructions

📦 Dataset Contents

This release contains 72 curated instances:

• —Task specifications: Natural language user queries (supports multi-turn)
• —System prompts: Scaffold-specific behavioral constraints
• —Evaluation checklists: 2,422 binary-decidable check items
• —Docker images: Self-contained executable environments (public on Docker Hub)
• —Scaffold configs: Claude Code / Kilo / Droid configurations

🐳 Docker Environments

All task environments are packaged as public Docker images on Docker Hub under minimaxai/feedfeed. You can pull and inspect any environment:

# Pull an environment image
docker pull minimaxai/feedfeed:<tag>

# Explore the workspace
docker run -it --rm minimaxai/feedfeed:<tag> /bin/bash

📊 Dataset Statistics

By Primary Category (the main instruction source being tested):

By Scaffold:

📝 Data Format

Each instance is a JSON object with the following fields:

{
  "instance_id": "md-course-builder-conventional-commits",
  "user_query": ["Implement the feature as specified..."],
  "system_prompt": "You are a CLI assistant...",
  "category": "Claude.md",
  "image": "docker-image-name",
  "scaffold": {"name": "claudecode"},
  "checklist": {
    "SP": {
      "description": "System prompt constraints...",
      "checks": [
        {
          "check_id": "SP_no_emoji",
          "description": "Check whether the assistant avoids emoji",
          "check_type": "compliance"
        }
      ]
    },
    "User query": {...}
  }
}

💻 Usage

1. Load the Dataset

from datasets import load_dataset

# Load the dataset
dataset = load_dataset("MiniMaxAI/OctoCodingBench")

# Filter by category
skill_tasks = [d for d in dataset["train"] if d["category"] == "Skill"]

# Filter by scaffold
claudecode_tasks = [d for d in dataset["train"] if d["scaffold"]["name"] == "claudecode"]

2. Evaluation Pipeline

The evaluation consists of three steps:

⚠️ Note: The complete evaluation scripts are under active development and will be open-sourced soon. Stay tuned for updates.

⚖️ Evaluation Metrics

🗓️ Roadmap

• —[x] Task Specifications, Checklists & Docker Environments — Released January 2026
• —[ ] Evaluation Code — Trajectory collection & LLM-as-judge scoring (Coming soon)

🏆 Leaderboard

📜 Citation

@misc{octocodingbench2026,
  title={OctoCodingBench: Instruction-Following Benchmark for Coding Agents},
  author={MiniMax},
  year={2026},
  publisher={Hugging Face}
}