ErtasAI/qmsum-summarizer-segenc-406m-spans

QMSum locate-then-summarize: span-trained SegEnc (406M)

A 406M Segment Encoder (Fusion-in-Decoder) fine-tuned to summarize retrieved transcript spans rather than a full meeting transcript. It is the artifact behind the paper's central result, which concerns alignment between the input a model is trained on and the input it receives at inference.

Through our inference port, the released Socratic SegEnc checkpoint loses 6.30 ROUGE-1 when handed 2,000-word retrieved spans in place of the capped long transcript it was built for, which puts it below our 1.2B system. Fine-tuned for that span regime on data verified byte-identical to our own summarizer's, it recovers the loss. On the test split, QMSum does not statistically separate it from our 1.2B on any reported metric, and crossing intervals do not establish equivalence. It uses about one-third as many total parameters and less than half the peak inference memory, on roughly a sixth of the input budget it was originally built for.

This is part of the release for the paper *Retrieved-Span Training for Efficient Query-Focused Meeting Summarization on QMSum* (Edward Xi Yang, Ertas AI; arXiv:2609.25028).

The release

Model details

• —Base model: Salesforce/socratic-pretraining-qmsum (Pagnoni et al., ACL 2023), a question-driven pretrained Segment Encoder already fine-tuned on QMSum
• —Architecture: Segment Encoder / Fusion-in-Decoder over BART-large. The transcript is split into overlapping fixed-length segments, each encoded independently, and the decoder cross-attends over the concatenated segment encodings
• —Parameters: 406,290,432, measured
• —Weights: model.safetensors, 1,625,463,896 bytes, float32
• —Training data: the same span-regime targets as our 1.2B summarizer, verified byte-identical, built from QMSum's own spans and references. No synthetic data is used anywhere in this project
• —Chunking: max_num_chunks 8, which is 4,008 tokens of capacity and covers the 2,000-word inference budget and the 3,000-word training source without truncation

Training, in two stages

Both stages ran on one NVIDIA RTX 5070 Ti (16 GB), about 45 minutes of GPU time in total.

The released weights are stage 2, epoch 4, which is effective epoch 8. It is the last checkpoint, chosen with no selection: the validation-best epoch was rejected as selection on noise. The learning-rate schedule restarts on resume, so this is a continued fine-tune rather than one clean 8-epoch run. The loss curve for stage 2 ships as history.json.

Results, full official QMSum test split

n=281, greedy decoding, one frozen scorer, a single test touch. Retrieval is identical for both rows: the promoted locator at 375-word windows, packed to a 2,000-word budget.

The difference is +0.93 ROUGE-1 with a 95% paired bootstrap interval of [-0.42, +2.24]. The meeting-cluster interval, which keeps each meeting's queries together, is [-0.27, +2.22]. Every reported metric's interval crosses zero: ROUGE-1, ROUGE-2 and ROUGE-Lsum point toward this model, ROUGE-L and BERTScore toward ours. QMSum does not statistically separate the two systems, and crossing intervals do not establish equivalence. The measured difference is resource use: about one-third as many total parameters (both counts include the shared locator) and less than half the peak inference memory.

Intended use and limitations

Research use, for reproducing and extending the results in the paper.

• —Scope. It summarizes located spans for a query. Given a full untruncated transcript it is outside the regime it was trained for, which is the entire point of the experiment.
• —Metrics. The results above are reference-anchored ROUGE and BERTScore, which do not measure whether individual propositions match the reference. In concurrent work of ours, fact-level scoring ranks proprietary hosted baselines above our 1.2B system, the reverse of their order on these metrics. That axis is not measured for this checkpoint.
• —Run-to-run variance. Reseeding the 1.2B summarizer moves its ROUGE-1 by 1.59 points, and this benchmark's detection floor is roughly one ROUGE-1 point at full split size. Read the +0.93 with both in mind.
• —The port caveat. Our port of the authors' inference code sits 3.3 ROUGE-1 below their own released predictions. Comparisons against the authors' full-input pipeline are scoped accordingly in the paper; this checkpoint's rows are internally consistent because every one of them runs through the same port.

Licence and attribution

This model is a modified version of Salesforce/socratic-pretraining-qmsum, fine-tuned by Ertas AI on retrieved-span data as described above. The base model is distributed under the BSD 3-Clause licence, and this derivative is released under the same licence. The full licence text, including the conditions and the disclaimer, ships in this repository as LICENSE.

Copyright (c) 2021, Salesforce.com, Inc. All rights reserved.

Salesforce's model card states that its release is "for research purposes only in support of an academic paper." We record that here alongside the BSD 3-Clause tag the repository carries, and our own use is academic research accompanying a paper.

Neither the name of Salesforce.com nor the names of its contributors are used to endorse or promote this derivative. The base model's authors have no involvement in this work.

Underlying data: QMSum is MIT licensed and draws on the AMI and ICSI meeting corpora (CC BY 4.0) and parliamentary committee proceedings. This repository redistributes no transcript text.

Citation

If you use this checkpoint, please cite the paper, and the base model's authors and QMSum alongside it:

@misc{yang2026retrievedspan,
  title         = {Retrieved-Span Training for Efficient Query-Focused Meeting Summarization on {QMSum}},
  author        = {Yang, Edward Xi},
  year          = {2026},
  eprint        = {2609.25028},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  doi           = {10.48550/arXiv.2609.25028},
  url           = {https://arxiv.org/abs/2609.25028},
}

@inproceedings{pagnoni2023socratic,
  author    = {Artidoro Pagnoni and Alexander R. Fabbri and Wojciech Kry\'{s}ci\'{n}ski and Chien-Sheng Wu},
  title     = {Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization},
  booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics},
  year      = {2023},
  note      = {ACL Anthology 2023.acl-long.713. arXiv:2212.10449},
}

@inproceedings{zhong2021qmsum,
  author    = {Ming Zhong and Da Yin and Tao Yu and Ahmad Zaidi and Mutethia Mutuma and Rahul Jha and Ahmed Hassan Awadallah and Asli Celikyilmaz and Yang Liu and Xipeng Qiu and Dragomir Radev},
  title     = {{QMSum}: A New Benchmark for Query-based Multi-domain Meeting Summarization},
  booktitle = {Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  pages     = {5905-5921},
  year      = {2021},
  note      = {ACL Anthology 2021.naacl-main.472. arXiv:2104.05938},
}