recube-anon-2026/recube-data

This dataset serves as the official data repository for the ReCUBE benchmark; the corresponding evaluation codebase and execution instructions can be found at https://anonymous.4open.science/r/ReCUBE-E0FB/README.md.

Data

This directory contains all benchmark data for ReCUBE.

Download

All data files are hosted on Hugging Face and can be downloaded using:

# Install huggingface_hub if not already installed
pip install huggingface_hub

# Download the entire dataset
huggingface-cli download recube-anon-2026/recube-data --repo-type=dataset --local-dir data/

# Or download in Python
from huggingface_hub import snapshot_download
snapshot_download(repo_id="recube-anon-2026/recube-data", repo_type="dataset", local_dir="data/")

Alternatively, you can clone the repository:

git clone [https://huggingface.co/datasets/recube-anon-2026/recube-data](https://huggingface.co/datasets/recube-anon-2026/recube-data) data/

Graphs (graphs/)

Dependency graphs in NetworkX format (.pkl files) for each functional subset.

Usage: Used by the graph-guided setting (bash_only_graph_setting) to provide efficient context navigation tools.

Format: NetworkX MultiDiGraph serialized with pickle

• —Node Types:
• —Directory nodes (e.g., /, app/, app/agent/)
• —File nodes with full source code (e.g., app/agent/base.py)
• —Class definitions with code snippet (e.g., app/agent/base.py:BaseAgent)
• —Function/method definitions with code snippet (e.g., app/agent/base.py:initialize_agent)

• —Node Attributes:
• —Node type (directory/file/class/function)
• —Full source code (for files) or code snippet (for classes/functions)
• —Line number range (for classes/functions only)

• —Edge Types:
• —contains: Hierarchical containment (directory→file, file→class, class→method, file→function)
• —imports: Import dependencies (file→imported entity)
• —invokes: Function call dependencies (function→called function/class)
• —inherits: Class inheritance (child class→base class)

• —Node ID Format:
• —Directories: Path string (e.g., app/agent)
• —Files: File path (e.g., app/agent/base.py)
• —Classes: {filepath}:{classname} (e.g., app/agent/base.py:BaseAgent)
• —Methods: {filepath}:{classname}.{method_name} (e.g., app/agent/base.py:BaseAgent.run)
• —Functions: {filepath}:{functionname} (e.g., app/utils.py:setup_logger)

Files: One graph per functional subset (e.g., 0.pkl, 1.pkl, ..., 54.pkl)

Prompts (prompts/)

Repository context files in text format (.txt) for each functional subset. Formatted repository context including:

• —README documentation
• —Package dependencies
• —All implemented files with complete code

Usage: Provided as input context to prompt experimental settings

Format: Structured text with XML-like tags:

<readme>
... repository documentation ...
</readme>

<dependencies>
... pip package list with versions ...
</dependencies>

<implementations>
## path/to/file.py

... complete implementation ...

</implementations>

Files: One prompt file per functional subset (e.g., 0.txt, 1.txt, ..., 54.txt)

Testcases (tests/)

Test files for evaluating generated code. These files are already embedded in Docker images.

Content: pytest test files for each target file

• —Unit tests for individual functions/classes
• —Integration tests for multi-component interactions
• —Real usage patterns extracted from original repositories

Structure:

tests/
├── 0/                          # Functional subset 0
│   ├── app_agent_base_test.py
│   ├── app_agent_manus_test.py
|   ├── ...
│   └── test_metadata.json
├── 1/                          # Functional subset 1
│   └── ...

Note: These files are copied into Docker images and do not need to be manually managed during evaluation.

test_metadata.json

{
  "repo_id": 0,
  "repo_url": "https://github.com/...",
  "tests_generated": "2026-01-05",
  "instances": [
    {
      "file": "app/agent/base.py",
      "test_file": "app_agent_base_test.py",
      "total_tests": 47,
      "functions_tested": 8,
      "validation_status": "pending",
      "real_usage_sources": { ... },
      "test_categories": {
        "unit_tests": 44,
        "integration_tests": 3
      },
      "difficulty_factors": [ ... ]
    }
  ],
  "validation": {
    "status": "passed",
    "tests_passed": 282,
    "tests_failed": 0
  }
}

Usage: Loaded during evaluation to map source files to test files and count expected tests.

Test Classifications (test_classifications/)

External vs internal test classifications for analyzing model capabilities.

Structure: One JSON file per functional subset (e.g., 0.json, 1.json)

Content:

{
  "repo_id": 0,
  "files": {
    "app/agent/base.py": {
      "test_file": "app_agent_base_test.py",
      "test_classifications": {
        "test_initialize_agent": "internal",
        "test_state_context": "external",
        "test_update_memory": "external",
        ...
      }
    }
  }
}

Test Types:

• —External: Tests that import and use the target file (API contract tests)
• —Internal: Tests that are within the target file itself (implementation tests)

Usage: Used during evaluation to calculate external/internal pass rates separately.

Target Files

target.json

366 verified target files used for official evaluation (i.e., 40 functional subset, 366 instances).

Format:

[
  {
    "repo_id": 0,
    "selected_files": [
      "app/agent/base.py",
      "app/agent/manus.py",
      ...
    ]
  },
  ...
]

Usage: Used by generation and evaluation scripts to determine which files to process.

Functional Subset Breakdown

Detailed breakdown of the 40 functional subsets in our benchmark. Each entry represents a specific functional subset of a larger repository, selected to target distinct development capabilities.

Diversity:

• —20 unique base repositories & 40 functional subsets
• —Domains: AI agents, ML training, OCR, TTS, RAG, code editing, MCP servers