Modelpublic

sandeep123/ML-LaySum-Qwen3-1.7B-SFT

sourceHugging Faceapache-2.0updated 1d agoView on Hugging Face

0likes156downloads

Model Card

ML-LaySum Qwen3-1.7B SFT

A full-parameter supervised fine-tuning of Qwen3-1.7B for English lay summarization of machine-learning research. It takes the abstract, introduction, and conclusion of a paper and generates a summary for a non-specialist reader. Generation uses Qwen3's non-thinking mode.

Training

Setting	Value
Base model	Qwen/Qwen3-1.7B (chat checkpoint)
Base revision	70d244cc86ccca08cf5af4e1e306ecf908b1ad5e
Dataset	ML-LaySum; ICML 2025 and 2026 papers and lay summaries
Dataset revision	a983411ae5100d8805f2852e10ebd43647001586
Training / validation / held-out test	6,911 / 879 / 852
Source	Abstract + introduction + conclusion, where available
Target	Reference lay summary
Objective	Cross-entropy on summary tokens and the turn-ending token; prompt tokens masked
Training	Full-parameter SFT, 3 epochs, 1,296 optimizer steps
Learning rate	2e-5; cosine schedule; 3% warmup
Optimizer	AdamW; weight decay 0
Effective batch size	16 (8 GPUs × 1 example × 2 accumulation steps)
Precision	FP32 weights; BF16 mixed-precision computation
Hardware	8 AMD MI210 GPUs
Context	Maximum configured length 32,768 tokens; no truncation or packing
Seed	42
Selection	Last checkpoint, step 1,296; not selected by best validation loss

Training resumed at step 1,000 after a storage-quota interruption, retaining optimizer, scheduler and random states. Test data were not used for training or checkpoint selection. Missing conclusion sections remain missing; no synthetic conclusions were inserted.

Usage

Install a Qwen3-compatible Transformers version (training used 4.57.6), PyTorch and Accelerate.

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "sandeep123/ML-LaySum-Qwen3-1.7B-SFT"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype=torch.bfloat16, device_map="auto"
)
paper_text = "Abstract: ...\n\nIntroduction: ...\n\nConclusion: ..."
instruction = (
    "Write a lay summary of the following machine learning research for a "
    "non-specialist reader. Explain the problem, main approach, and findings in "
    "clear language, faithfully using only the supplied paper text. Return only "
    "the summary.\n\nPaper text:\n"
)
text = tokenizer.apply_chat_template(
    [{"role": "user", "content": instruction + paper_text}],
    tokenize=False, add_generation_prompt=True, enable_thinking=False,
)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
with torch.inference_mode():
    output = model.generate(**inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(output[0, inputs.input_ids.shape[1]:], skip_special_tokens=True))

The decoding settings above are an example, not a validated optimal configuration. Uploaded weights are FP32 (approximately 6.9 GB); loading in BF16 reduces weight memory to approximately 3.4 GB, excluding activations and the generation cache.

Evaluation and limitations

No held-out generation benchmark or human evaluation has yet been completed. This is an SFT baseline, with no claim of state-of-the-art performance. Outputs can omit qualifications, misstate findings or introduce unsupported explanations. Check summaries against the paper before publication. Validation cross-entropy is a training diagnostic, not a measure of lay-reader comprehension. See training_summary.json for validation losses.

Files and license

This repository contains inference weights, tokenizer files and sanitized training metadata. Optimizer states, account credentials, raw training records and private cluster paths are not uploaded. Model weights are released under Apache 2.0, following the base model license. Dataset and source-paper terms are separate; see the dataset card. Dataset access may require permission independently of this public model.