Datasetpublic

rajatagarwal457/sarvam-top10-review

Sarvam Top-10 Task Review 10 tasks from the 118-task gworkspace_hard_v1 benchmark, selected for trace richness (actions per goal ≥ 1.5). 5 tasks where Sarvam-105B passed; 5 engaged failures. Each row is a Sarvam-105B rollout (n=1); the other three models' results live in the peer_results column. Selection Passes (5): all Sarvam-105B passes in the dataset, per the trace file. Failures (5): Sarvam ≥7 actions, failed, capped at 2 per family for diversity.… See the full description on the dataset page: https://huggingface.co/datasets/rajatagarwal457/sarvam-top10-review.

sourceHugging Facecc-by-4.0updated 4mo agoView on Hugging Face

1likes7downloads

Dataset Card

Sarvam Top-10 Task Review

10 tasks from the 118-task gworkspace_hard_v1 benchmark, selected for trace richness (actions per goal ≥ 1.5). 5 tasks where Sarvam-105B passed; 5 engaged failures. Each row is a Sarvam-105B rollout (n=1); the other three models' results live in the peer_results column.

Selection

—Passes (5): all Sarvam-105B passes in the dataset, per the trace file.
—Failures (5): Sarvam ≥7 actions, failed, capped at 2 per family for diversity.

Failure-mode taxonomy

code	meaning	primary predicate it breaks
`empty_doc_share`	Doc created and shared but never populated (no `docs.batchUpdate`)	`doc_contains(non_empty=True)`
`wrong_reply_method`	Used `messages.send` instead of `messages.reply` for a thread reply	`reply_on_thread`
`missing_label`	Replied but never applied the required label	`message_has_label`
`wrong_calendar_attrs`	Event inserted with wrong attendees / duration / window	`event_exists`
`asymmetric_share`	Shared with one recipient but skipped others	`file_shared`
`missing_followup_task`	Prompt asked for a follow-up Task; agent never called `tasks.insert`	`task_exists`
`wrong_recipient`	Replied / shared with the wrong person	recipient-specific
`ask_loop`	Called `ask` repeatedly on routine choices; burned the action budget	diffuse
`act_when_should_ask`	Acted on an under-specified prompt instead of asking	`asked_for_clarification`
`early_termination`	Emitted `{}` or `done` before completing the workflow	diffuse
`hallucinated_api`	Invented a non-existent method name	(single wasted step)
`prompt_ambiguity`	Even the ceiling model fails — failure traces to the prompt, not the model	varies

Tags are not mutually exclusive. A single trajectory can carry multiple tags.

Columns

column	type	description
`task_id`	string	unique id
`family`	string	task family
`register`	string	casual / canonical / structured / voice / terse
`n_goals`	int	number of verifier predicates
`prompt`	string	exact text sent to the model
`verifier_spec`	list	predicate calls evaluated against the final world state
`model_id`	string	always `sarvam105b` for now
`passed`	bool	all goals satisfied
`goals_met`	int	how many of n_goals satisfied
`n_actions`	int	tool calls in the rollout
`trajectory`	list	per-step thought + action + observation + state_delta
`goal_results`	list	per-predicate satisfied flag
`primary_failure_mode`	string	single most diagnostic tag (filterable)
`failure_modes`	list[string]	multi-label tags from the taxonomy above
`failure_mode_evidence`	dict	for each tag, the trajectory steps that justify it
`is_<tag>`	bool	one per tag in the taxonomy — use these for filtering in the viewer
`task_level_failure_mode`	string	dominant failure pattern across all 4 models on this task
`task_level_summary`	string	one-paragraph explanation of the task's typical failure pattern
`peer_results`	dict	gpt-5.2 / gpt-oss / glm results on the same task
`split`	string	"pass" or "fail"
`review_notes`	string	per-row analysis

How to view

Open the Dataset Viewer. Filter failure_modes contains empty_doc_share to see every row exhibiting that pattern. Click a row to expand trajectory and see the model's reasoning step by step.

Notes on data quality

—The dataset summary file gworkspace_hard_v1.json had 5 stale Sarvam pass/exec rows that disagreed with the trace file. This dataset uses the trace file as source of truth.
—Failure-mode tags were generated by heuristic rules and reviewed manually. Where the ceiling model (gpt-5.2) also fails a predicate, the tag prompt_ambiguity is added — those are candidates for prompt rewrites rather than model evaluation.