Yakk99/scihigh2026-subtask1-bart-large-cnn

scihigh2026-subtask1-bart-large-cnn

Fine-tuned `facebook/bart-large-cnn` for SciHigh-2026 Subtask 1: Research Highlight Generation — generating 3-5 short, telegraphic highlight sentences (~50-55 words total) from a scientific paper's abstract.

Task

Given a paper's abstract, generate a small set of highlight sentences that summarize the paper's key contributions/findings in a compressed, telegraphic style (not a paraphrase-style abstractive summary). Ranked by ROUGE-L F1 (primary metric), with ROUGE-1/2, METEOR, and BERTScore also reported.

Baseline: the FIRE-2025 shared-task's winning submission (fine-tuned Pegasus-large + NER features, 10 epochs) scored 23.45% ROUGE-L F1.

Training data

Fine-tuned on the MixSub-SciHigh dataset, an expanded pool of 15,960 (Abstract, Highlights) pairs: the official 2026 task's 10,000-row training split, plus 5,960 additional real training pairs recovered from the dataset's original source release (leakage-checked by exact Abstract-text match against the official validation/test splits before being used for training — zero overlap confirmed). Validated against the official held-out validation split (1,985 rows), which was not used for training, only for per-epoch checkpoint selection.

Training recipe

Hyperparameters mirror the FIRE-2025 baseline recipe unchanged (a hyperparameter sweep over learning rate, warmup, and label smoothing found every variant flat-to-worse on top of this backbone). The epoch ceiling is a safety bound, not a fixed schedule — early stopping determines the actual stopping point.

In the training run that produced this checkpoint, validation ROUGE-L peaked after epoch 1 and did not improve in epochs 2 or 3; early stopping halted training after epoch 3 (before reaching the 4-epoch ceiling), and the epoch-1 checkpoint (this one) was restored as the final model.

Evaluation results

Evaluated on the full official validation split (1,985 rows):

Usage

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model_id = "Yakk99/scihigh2026-subtask1-bart-large-cnn"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id)

abstract = "..."  # a scientific paper's abstract
inputs = tokenizer(abstract, max_length=512, truncation=True, return_tensors="pt")
output = model.generate(**inputs, max_length=100, num_beams=4)
print(tokenizer.batch_decode(output, skip_special_tokens=True)[0])

max_length=100 is set explicitly to match the recipe's output-length budget — the checkpoint's own generation_config already carries this value, so a plain model.generate(**inputs) call (with no extra arguments) also reproduces the reported results.

Files

• —train_final.py / train_final.ipynb — the standalone training script and a companion notebook documenting this exact recipe end-to-end (data, model, training configuration, decode settings, and the real reported results above).

Limitations

• —Abstracts in the source dataset are frequently truncated mid-sentence (a scraped-preview artifact in the original data release), which imposes a hard ceiling on achievable ROUGE-L for the affected subset — some reference highlights cite facts genuinely absent from the (truncated) input.
• —This model was selected and evaluated on ROUGE-L against a single validation split; it has not been evaluated for factual accuracy/ hallucination beyond standard n-gram/embedding-based metrics.