Datasetpublic

narcolepticchicken/agent-cost-traces

Agent Cost Traces: Synthetic Training Data 10,000 synthetic agent traces for training cost-aware model routers and agent optimizers. Schema Field Type Description trace_id string Unique identifier request string User request text task_type string One of 9 task categories difficulty int Estimated difficulty (1-5) model_tier int Model tier used (1-5) model_success bool Whether the model succeeded optimal_tier int Minimum tier that would… See the full description on the dataset page: https://huggingface.co/datasets/narcolepticchicken/agent-cost-traces.

sourceHugging Facemitupdated 5mo agoView on Hugging Face

0likes52downloads

Dataset Card

Agent Cost Traces: Synthetic Training Data

10,000 synthetic agent traces for training cost-aware model routers and agent optimizers.

Schema

Field	Type	Description
trace_id	string	Unique identifier
request	string	User request text
task_type	string	One of 9 task categories
difficulty	int	Estimated difficulty (1-5)
model_tier	int	Model tier used (1-5)
model_success	bool	Whether the model succeeded
optimal_tier	int	Minimum tier that would succeed
tool_calls	list	Tool calls made
verifier_called	bool	Whether a verifier was called
retries	int	Number of retries
final_success	bool	Whether the task ultimately succeeded
total_cost	float	Total cost of the run
failure_tags	list	Tags describing failures
cheaper_alternative	int/null	Cheaper tier that would also work
recovery_action	string	What recovery was attempted
verifier_needed	bool	Whether verification was needed
escalation_needed	bool	Whether a higher tier was needed

Task Types and Difficulty

Task Type	Base Difficulty	Floor Tier
quick_answer	1	1
document_drafting	2	2
tool_heavy	2	2
retrieval_heavy	2	2
research	3	3
coding	3	3
unknown_ambiguous	3	3
long_horizon	4	3
legal_regulated	5	4

Success Model

P(success) = tier_strength^(difficulty × 0.6)

Tier strengths: tiny=0.35, cheap=0.55, medium=0.80, frontier=0.93, specialist=0.97

Use Cases

—Train model cascade routers
—Evaluate cost-quality frontiers
—Benchmark cost-aware agent optimizers
—Study failure recovery policies
—Research selective verification strategies

⚠️ Limitations

This is synthetic data. Real execution data produces dramatically better routers. On SWE-bench, the v8 router trained on this synthetic data increased cost by 11.6%, while v10 trained on just 500 real outcomes saved 23.3%. Use this data for prototyping and sanity checks, not for production routing.

For real execution data, see:

License

MIT