jumplander/JL-AgentBehavior-10K

JL-AgentBehavior-10K

    

JL-AgentBehavior-10K is a 10,000-record, English-language research-preview dataset for studying and training the behavioral policy of repository-level coding agents.

The dataset does not treat a coding agent as a chatbot that maps a request directly to a block of code. It represents an agent as a policy operating across a sequence of observable decisions:

task
  -> repository evidence
  -> bounded plan
  -> tool selection
  -> scoped edit strategy
  -> verification
  -> failure diagnosis and repair
  -> evidence-based final report

Each record is labeled by the behavior it is intended to influence, the type of supervision it provides, its provenance, and its verification level. The release includes trajectory supervision, chosen/rejected preference pairs, and failure-repair examples across eight programming-language ecosystems.

[!IMPORTANT] Version 1.0.0 is a deterministically generated synthetic policy corpus. Its schema, counts, identifiers, fingerprints, and repository-family split isolation are validated. The repositories are not materialized and the candidate patches or expected tool observations are not runtime-executed. Every record states execution_verified: false and quality_tier: silver_structural. Do not describe this release as a runtime-verified or human-reviewed Gold dataset.

Why this dataset exists

Many code instruction datasets supervise only the final answer. That is useful for code completion and isolated problem solving, but it leaves important agent behaviors underspecified:

• —whether the model searches before editing;
• —whether it grounds file and symbol references in repository evidence;
• —whether the plan is proportional to the task;
• —whether the selected tool and arguments are valid;
• —whether the patch remains inside the requested boundary;
• —whether tests are executed before a success claim;
• —whether a failed attempt changes the hypothesis;
• —whether an operation requires user approval;
• —whether the final report distinguishes evidence from assumption.

JL-AgentBehavior-10K makes those behaviors explicit and measurable. It is intended as a controlled starting point for supervised fine-tuning, preference optimization, ablation studies, data-pipeline development, and agent-policy research.

It is not intended to replace repository execution, real issue/patch trajectories, hidden tests, human code review, or an independent evaluation benchmark.

Release status and evidence level

The evidence level is encoded per record rather than implied by the dataset title. This is intentional: users should be able to filter by provenance when future verified releases are merged with this corpus.

Intended behavioral effects

The dataset is organized around nine primary behavioral axes. Each axis is associated with a measurable hypothesis rather than a general claim that the agent becomes “better.”

These effects are hypotheses. They must be tested against a base model and appropriate controls. The dataset itself does not establish causality.

Dataset composition

Splits

Records from one synthetic repository family appear in only one split. This prevents direct family leakage within the packaged splits. The test split is suitable for pipeline checks, but it should not be treated as a strong independent benchmark because it is generated by the same scenario system as the training data.

Supervision formats

Task distribution

Difficulty distribution

Difficulty is a generation label based on scenario composition. It is not calibrated against human completion time or model pass rate in v1.

Language and code coverage

All natural-language instructions and supervision annotations are in English.

The command strings are part of synthetic environment descriptions. They are not evidence that the referenced commands were executed.

Record model

Each row is a JSON object with five conceptual layers:

1. 1.Task contract — instruction, constraints, and acceptance criteria.
2. 2.Environment description — repository family, target path, test path, symbols, and candidate commands.
3. 3.Behavioral labels — intended policy, approval boundary, and expected tool order.
4. 4.Supervision — trajectory, preference pair, or repair trace.
5. 5.Evidence metadata — provenance, quality tier, execution status, and fingerprint.

Compact example

{
  "id": "jlab-v1-00001",
  "version": "1.0.0",
  "split": "train",
  "record_type": "trajectory",
  "primary_behavior": "repository_grounding",
  "secondary_behaviors": [
    "tool_selection_execution",
    "bounded_code_editing"
  ],
  "language": "typescript",
  "task_type": "bug_fix",
  "difficulty": "medium",
  "task": {
    "instruction": "Fix the defect where the session returns stale state after an update in the identity gateway.",
    "constraints": [
      "Preserve the public API",
      "Do not add dependencies",
      "Limit changes to relevant files"
    ],
    "acceptance_criteria": [
      "The session no longer exhibits the specified behavior",
      "The public contract remains compatible",
      "A targeted verification path is identified",
      "The final change remains inside the requested scope"
    ]
  },
  "environment": {
    "repository_id": "jl-synrepo-0000",
    "target_file": "src/session/session_000_00.ts",
    "test_file": "test/session/test_session_000_00.ts",
    "targeted_test_command": "npm test -- test/session/test_session_000_00.ts"
  },
  "behavioral_labels": {
    "must_ground_before_edit": true,
    "must_verify_before_success_claim": true,
    "requires_approval": false,
    "target_behavior": "Ground every decision in repository evidence before editing."
  },
  "provenance": {
    "origin": "synthetic",
    "repository_materialized": false,
    "execution_verified": false,
    "human_reviewed": false,
    "release_stage": "research_preview"
  },
  "quality": {
    "schema_validated": true,
    "execution_verified": false,
    "quality_tier": "silver_structural"
  },
  "supervision": {
    "policy_summary": "Ground every decision in repository evidence before editing.",
    "steps": [
      {
        "step": 1,
        "phase": "ground",
        "action": {
          "tool": "search",
          "arguments": {
            "query": "SessionPolicy00000|cache_invalidation_defect",
            "path": "."
          }
        }
      }
    ]
  },
  "fingerprint": "<sha256>"
}

The actual rows contain the complete supervision object. See `schema.json` and `docs/SCHEMA.md` for field-level details.

Tool vocabulary

The release uses a small, implementation-neutral action vocabulary:

• —search
• —list_directory
• —read_file
• —update_plan
• —apply_patch
• —run_tests
• —run_linter
• —git_diff
• —diagnose_failure
• —review_diff
• —request_approval
• —continue_without_approval

These are abstract training actions, not a claim about a specific agent runtime. Before training, map them to the exact tool names, role boundaries, JSON schemas, and result formats used by the deployment harness. A mismatch between training-time and inference-time tool interfaces can reduce transfer.

Loading the dataset

Hugging Face Datasets

After publishing under the intended organization:

from datasets import load_dataset

dataset = load_dataset("jumplander/JL-AgentBehavior-10K")
print(dataset)
print(dataset["train"][0]["primary_behavior"])

Local JSONL

from datasets import load_dataset

dataset = load_dataset(
    "json",
    data_files={
        "train": "data/train.jsonl",
        "validation": "data/validation.jsonl",
        "test": "data/test.jsonl",
    },
)

Stream records

import json

with open("data/train.jsonl", encoding="utf-8") as handle:
    for line in handle:
        record = json.loads(line)
        if record["primary_behavior"] == "failure_diagnosis_repair":
            print(record["id"], record["record_type"])

Recommended training use

1. Start with a code-capable base model

This dataset is designed to influence control policy. It is too small and too synthetic to create repository-level coding ability from scratch. The base model should already understand code, common software-engineering concepts, and the relevant programming languages.

2. Preserve provenance during preprocessing

Do not discard these fields before deciding how the record is used:

• —record_type
• —primary_behavior
• —provenance.execution_verified
• —provenance.human_reviewed
• —quality.quality_tier

Future Gold records should be upweighted relative to synthetic policy records only after evaluation confirms the weighting is useful.

3. Do not train on synthetic tool results as facts

Tool outputs and expected observations describe the intended policy structure. They are not executed evidence. A conservative training formatter should:

• —mask the user task and environment from the loss;
• —mask external tool results from the loss;
• —apply loss to assistant decisions, tool calls, repair classifications, and final-report contracts;
• —avoid teaching the model to fabricate a successful test result;
• —preserve the distinction between expected, simulated, and executed evidence.

4. Separate training objectives by record type

The repository includes scripts/format_sft.py as a conservative reference converter. It is not a universal chat template; adapt it to the tokenizer and agent runtime being trained.

5. Mix with real execution data

For serious coding-agent training, combine this release with legally usable, deduplicated data containing:

• —materialized repository snapshots;
• —issue or task provenance;
• —actual tool outputs;
• —applicable patches;
• —executed targeted and regression tests;
• —review outcomes;
• —explicit license records;
• —contamination controls.

Use execution_verified and quality_tier to prevent synthetic and executed records from being silently mixed.

Evaluation protocol

Do not evaluate the model only on the packaged synthetic test split. Use unseen, independently constructed, executable repository tasks.

Core metrics

Minimum controlled comparison

Compare at least:

1. 1.the unchanged base model;
2. 2.the base model fine-tuned on a random 10K code-instruction control;
3. 3.the base model fine-tuned on the complete JL-AgentBehavior-10K mixture;
4. 4.behavior-specific ablations.

Recommended ablations:

full mixture
full minus repository_grounding
full minus planning_decomposition
full minus tool_selection_execution
full minus test_verification
full minus failure_diagnosis_repair
full minus permission_scope_safety

Run models in the same harness with the same tool schemas, budgets, decoding settings, and repository tasks. For paired binary outcomes, report confidence intervals and use a paired method such as bootstrap resampling or McNemar’s test where appropriate. Report adverse shifts, not only average task success.

Example falsifiable hypotheses

An effect should not be attributed to this dataset unless it survives an appropriate control and ablation analysis.

Quality assurance

The packaged validator checks:

• —exact total and split counts;
• —exact behavior, record-type, and language quotas;
• —required top-level fields;
• —unique IDs;
• —unique task instructions;
• —unique SHA-256 record fingerprints;
• —fingerprint integrity;
• —split field/file agreement;
• —repository-family isolation across splits;
• —explicit non-verified provenance;
• —expected quality tier.

Run:

python scripts/validate_dataset.py

Regenerate the deterministic release:

python scripts/generate_dataset.py
python scripts/validate_dataset.py --report validation_report.json
python scripts/make_checksums.py

Generation uses the fixed seed 6172026. Identical code and environment should reproduce identical JSONL records and fingerprints.

Data generation methodology

Version 1 is generated from controlled combinations of:

• —nine behavioral targets;
• —three supervision formats;
• —eight language ecosystems;
• —seven software-engineering task types;
• —three difficulty labels;
• —twenty-five synthetic service domains;
• —twenty-five component families;
• —twenty defect patterns;
• —bounded constraint sets;
• —behavior-specific chosen and rejected action patterns;
• —failure signals and repair sequences.

The generator produces 500 synthetic repository families with 20 tasks per family. Families are assigned to splits before records are written. Each record receives a content-derived SHA-256 fingerprint.

The methodology prioritizes controlled coverage and explicit labels over naturalistic repository realism. This makes the dataset useful for policy experiments and pipeline engineering, while limiting claims about real-world patch correctness.

Biases and limitations

Synthetic regularity

The data is compositionally generated. Models may learn repeated phrasing, common action order, or template artifacts instead of general agent behavior. Mixing with diverse real trajectories and running a template-detection audit are strongly recommended.

No executable repository state

Paths, symbols, commits, tool results, patches, and test commands describe synthetic environments. They are not backed by repository archives in v1. A model trained only on this release may learn the shape of verification without learning reliable execution.

No human preference study

Chosen/rejected pairs encode the dataset designers’ behavioral specification. They do not represent a broad developer preference survey or inter-annotator agreement study.

Tool vocabulary bias

The abstract tool interface favors search/read/patch/test loops. Agents using AST tools, language servers, database consoles, browser tools, distributed execution, or proprietary IDE actions need explicit interface adaptation.

English-only supervision

Task text and annotations are English. The dataset does not establish multilingual agent performance.

Unequal language coverage

Python and TypeScript dominate the mixture. Rust, C#, and C++ are intentionally smaller and should not be used to claim balanced language performance.

Safety coverage is narrow

The safety subset focuses on permission and scope boundaries in coding workflows. It is not a comprehensive cybersecurity, secure-code-generation, or alignment dataset.

Packaged test split is not an independent benchmark

Although repository families are split-disjoint, all records share one generation system. Use an independently sourced executable benchmark for reported model results.

Misuse and out-of-scope use

Do not use this dataset to:

• —claim that generated patches pass tests;
• —claim human review or production validation;
• —benchmark models on the same data used for training;
• —train an agent to execute destructive or external actions without runtime authorization controls;
• —replace secure sandboxing, access control, audit logging, or human review;
• —infer that a structurally valid trajectory is a correct solution to a real repository task.

Any deployed coding agent should enforce permissions and trust boundaries in the runtime. Dataset supervision is not a security boundary.

Roadmap to a Gold release

A future gold_executed tier should require:

1. 1.a licensed, materialized repository snapshot;
2. 2.an immutable before-state identifier;
3. 3.a real task or controlled bug injection;
4. 4.recorded tool calls and tool outputs;
5. 5.an applicable patch;
6. 6.executed targeted tests;
7. 7.executed regression tests where feasible;
8. 8.static-analysis results;
9. 9.a diff-scope audit;
10. 10.human review or adjudication;
11. 11.contamination and near-duplicate checks;
12. 12.an immutable evidence manifest.

Gold should be an evidence tier, not a marketing label.

Project files

JL-AgentBehavior-10K-v1/
├── README.md
├── LICENSE.md
├── CITATION.cff
├── VERSION
├── schema.json
├── stats.json
├── validation_report.json
├── checksums.sha256
├── data/
│   ├── train.jsonl
│   ├── validation.jsonl
│   └── test.jsonl
├── docs/
│   ├── BEHAVIOR_SPEC.md
│   ├── DATA_STATEMENT.md
│   ├── EVALUATION_PROTOCOL.md
│   ├── SCHEMA.md
│   └── TRAINING_GUIDE.md
└── scripts/
    ├── format_sft.py
    ├── generate_dataset.py
    ├── make_checksums.py
    └── validate_dataset.py

Versioning

This release follows semantic versioning for the dataset package:

• —Patch: documentation, metadata, or validation fixes that do not intentionally change record semantics.
• —Minor: additive fields, new compatible supervision views, or additional records.
• —Major: schema-breaking changes, changed behavioral taxonomy, or materially different generation/evidence policy.

Current version: 1.0.0.

License

The packaged dataset and original documentation are released under the Creative Commons Attribution 4.0 International (CC BY 4.0) license. See `LICENSE.md`.

Attribution should identify JumpLander and the dataset name/version. Users remain responsible for reviewing the license compatibility of any external data they combine with this release.

Citation

@dataset{jumplander_agentbehavior_10k_2026,
  author       = {JumpLander},
  title        = {JL-AgentBehavior-10K: Structured Behavioral Supervision for Coding Agents},
  year         = {2026},
  version      = {1.0.0},
  publisher    = {Hugging Face},
  url          = {https://huggingface.co/datasets/jumplander/JL-AgentBehavior-10K}
}

References

The release is informed by work on interleaved reasoning/action policies, repository-level software-engineering evaluation, and agent-computer interfaces. These references do not imply endorsement of or involvement in this dataset.

• —Yao et al. ReAct: Synergizing Reasoning and Acting in Language Models, 2022/2023.
• —Jimenez et al. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, 2023/2024.
• —Yang et al. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering, 2024.
• —Hugging Face. Dataset Cards documentation.

About JumpLander

JumpLander is an engineering project focused on AI-assisted software development, coding-agent loops, repository intelligence, specialized training and evaluation datasets, developer tooling, and secure execution infrastructure.

• —Website: jumplander.org
• —Hugging Face: huggingface.co/jumplander
• —GitHub: github.com/jumplander

This dataset is a research artifact under active development. Technical findings, limitations, and reproducible evaluation results are preferred over unsupported capability claims.