Eyerf/agentblackbox-rag-repair-outcomes

AgentBlackBox RAG Repair Outcome Dataset

This dataset contains replay-labeled repair outcome data for AgentBlackBox, a counterfactual debugging framework for language agents.

The data is built around failed RAG/document-recall agent traces, candidate repairs, counterfactual replay labels, and repair-ranking evaluation outputs.

Contents

datasets/ worldmodelrankerdatasetv2_train10k/ pointwise/ listwise/ stats.json

qwenpairwisesftv2train10kfixed/ train.jsonl dev.jsonl test.jsonl stats.json samplepreview.json

rawruns/ hotpotqavalidation20260610210103/ summary.jsonl repairoutcomes.jsonl summary.csv repairoutcomes.csv

hotpotqatraincombined10kv1/ summary.jsonl repair_outcomes.jsonl

evals/ worldmodelrankerbaselinev2train10k/ v2packageevalqwenpairwiser32e3logodds/ qwen38bpairwiseloraevalv2train10kr32e3_logodds/

results/ final_tables/

Dataset Summary

The dataset supports research on:

• —language-agent failure analysis
• —RAG repair generation
• —counterfactual replay evaluation
• —repair outcome modeling
• —pairwise repair ranking
• —world models for agent debugging

The main learning problem is:

failed trace + candidate repair(s) -> replay-verified repair outcome / repair ranking

Main Splits

The main ranker dataset is located at:

datasets/worldmodelrankerdatasetv2_train10k/

It includes pointwise and listwise formats for repair outcome modeling and repair ranking.

The Qwen pairwise SFT dataset is located at:

datasets/qwenpairwisesftv2train10k_fixed/

It includes train/dev/test JSONL files for pairwise repair ranking.

Evaluation Results

Final result tables are included in:

results/final_tables/

The main ranking table is:

results/finaltables/tablerankerperformancewide.csv

The mixed-trace analysis table is:

results/finaltables/tablemixedtraceanalysis.csv

Related Model

The Qwen3-8B LoRA pairwise repair ranker is available at:

Eyerf/agentblackbox-qwen3-8b-rag-repair-ranker-lora

Related Code

The AgentBlackBox code repository is available at:

https://github.com/eyerf/agentblackbox

Source and License Notes

This dataset is derived from RAG experiments based on HotpotQA-style question answering and AgentBlackBox-generated repair/replay outputs.

The released files contain derived failed-trace metadata, repair candidates, replay labels, pairwise ranking data, and evaluation outputs.

Please cite the original QA benchmark sources and AgentBlackBox when using this dataset.

Limitations

• —The dataset is focused on RAG/document-recall agent failures.
• —Replay labels depend on the AgentBlackBox replay/evaluation setup.
• —The dataset should not be interpreted as a general benchmark for all agent failures.
• —The pairwise ranking data reflects the candidate repair generation and replay policy used in the experiments.