ngong123/qmsum-test281-qwen3.5-4b-results

QMSum test split (281 pairs) — Qwen3.5-4B-UD-Q8KXL.gguf results

Benchmark results, not a benchmark. Every query/reference pair of the complete QMSum test split run through a file-reading agent, then graded three ways. Produced with `harness_bench`.

The source corpus (QMSum, Yale-LILY, MIT) is not redistributed here. Meetings are referenced by content fingerprint (transcript_file); gold reference summaries are omitted. Join against the upstream corpus if you need them.

What was run

Headline results

Over the 274 tasks that pass the gradeability screen (7 quarantined pairs are included but excluded from the headline):

Read these three caveats before citing anything

1. 1.As-shipped ROUGE-L sits at its own do-nothing floor. A canned generic paragraph written without reading any transcript scores ~0.136 against these references, versus the model's 0.164 — about 1.2× headroom. QMSum references are ~45% stopwords, so the LCS largely matches the/and/of/to. The stopword-stripped *_content columns have ~4× headroom and are the deterministic numbers worth using.
2. 2.The hallucination count is a count, not a rate — and is bimodal. 145/280 (52%) summaries were flagged completely clean; the mean is pulled by a thin tail. Distribution: {0: 145, 1: 70, 2: 39, 3: 15, 4: 8, 5: 2, 6: 1}. Counts of ≥3 are known to be inflated relative to hand-grading, so treat this as a binary screen (0–1 = clean) rather than a precise measure, and read unsupported_claims before trusting any high count.
3. 3.The judge is self-inconsistent. Three identical runs at temperature 0 agreed on only 5/20 tasks (mean spread 1.47 claims). A single-call judge number is not a point estimate. Coverage and hallucination are comparable only to a run using the same judge_model + judge_prompt with the same denominator — both are recorded per row.

Configs

scores — 281 rows

One row per task: task identity, the model's prediction, all six ROUGE variants plus the per-task canned floor, both judge axes, and the unsupported_claims list backing every hallucination count. hallucination_count == len(unsupported_claims) by construction, so every number here is auditable.

trajectories — 281 rows

The agent's behaviour per task: n_llm_calls, n_tool_calls, tools_used, token totals, and a steps list of tool calls and results (deduplicated; arguments truncated to 4,000 chars). Useful for read-strategy analysis — e.g. two tasks in this run originally failed by looping (113× rm summary.txt), which is visible here and not in the scores.

wire — 281 rows

Raw request/response records as JSON strings, one row per task. Provider-shaped and large; for debugging, not analysis.

Fields worth knowing about

• —query_shape — whole_meeting / summarize_scoped / targeted, assigned by a pre-registered frozen regex. 64 of the 244 corpus-specific queries literally begin "Summarize…", so the corpus general/specific label conflates scope with focus; this axis separates them.
• —screen_quarantine / screen_reason — a gradeability verdict from an offline screen (degenerate reference, near-duplicate reference, or an implausible do-nothing floor). Kept, flagged, excluded from the headline.
• —prediction_at_length_cap — the harness caps a saved answer at 2,000 chars. Such a summary may end mid-sentence, which inflates its hallucination count.
• —judge_budget_escalated — the provider overran its own reasoning cap and the call was retried with a larger completion budget. The count is usable but the call was not the standard one.

Sanitization

Internal lab infrastructure (SUT endpoint, corporate proxy hostnames, operator username and absolute local paths) is replaced with placeholders such as SUT_ENDPOINT. No API keys or credentials are present in any config.

Reproducing

scripts/run_qmsum.sh data/suite_qmsum_test281.json 4     # SUT only
scripts/judge_qmsum_run.sh runs/<tag> 8                  # ROUGE + both judge axes
python3 data/scripts/qmsum_run_report.py runs/<tag> --per-task