CoolFace
Datasetpublic

beyarkay/slash-bad

slash-bad Moments when a coding agent did something its user didn't like, flagged in the middle of real work with /bad <what went wrong>, then re-run on newer models. Each row is one flagged moment: the conversation up to that point, the response that annoyed the user, the user's own words about what was wrong with it, and blind verdicts on how far that complaint applies to the original response and to re-runs by other models. The data comes from slash-bad, a personal benchmark… See the full description on the dataset page: https://huggingface.co/datasets/beyarkay/slash-bad.

sourceHugging Facecc-by-4.0updated 2d agoView on Hugging Face
0likes42downloads
Dataset Card

slash-bad

Moments when a coding agent did something its user didn't like, flagged in the middle of real work with /bad <what went wrong>, then re-run on newer models.

Each row is one flagged moment: the conversation up to that point, the response that annoyed the user, the user's own words about what was wrong with it, and blind verdicts on how far that complaint applies to the original response and to re-runs by other models.

The data comes from slash-bad, a personal benchmark for Claude Code and Codex. People contribute their cases with /bad-share, which anonymises them and opens a pull request here.

Fields (data/*.jsonl)

fieldwhat it is
idrandom id for the case
harnessclaude-code or codex
harness_versionversion of the harness when the case was captured
modelthe model that wrote the flagged response
complaintwhat the user said was wrong, verbatim
flagged_responsethe response they flagged
conversationthe context before it, as steps: user, assistant, tool_call, tool_result
verdictsblind judge scores, 0 to 10, where 10 means the complaint doesn't apply; response is flagged or replay, with the replaying model and its text
statscounts: user messages, tool calls, and whether the flagged message came straight after a tool result

sessions/claude_code/<id>.jsonl and sessions/codex/<id>.jsonl hold the same context in the harness's own format, so a case can be replayed exactly.

How a replay works

Only the flagged message is regenerated. The stored conversation is resumed under the current harness, with its real system prompt and tools, and any tool call the new model tries is refused with a plain "permission denied" message. That keeps the comparison fair without copying the repository or mocking tool output, at the cost of not testing whether a newer model would have taken a different path earlier in the conversation.

Privacy

Contributions are anonymised automatically before upload: known secret formats, credentials in URLs, key/value secrets, email addresses, private IP addresses, home directory paths, the local username and git identity are replaced, and the user's CLAUDE.md is dropped. Code and file contents are not removed, and every contributor is asked to read their files before they upload. If you find something here that shouldn't be, open a discussion and it will be removed.

Seed data

The first cases were produced deliberately: sessions of claude-opus-5 on small throwaway repositories, in scenarios built to provoke the failure modes described in Common LLM failure modes, then flagged and compared against claude-opus-5-5.