assemsabry/Research-News-AI-Summarizer

Research & News AI Summarizer

[image]

Model Type: NLP / Text Summarization

Model Description

Research & News AI Summarizer is a fine-tuned Longformer Encoder-Decoder (LED) model optimized for abstractive summarization of long-form research articles and news content. It is trained using a custom 3-stage curriculum learning strategy across multiple versions of the CNN/DailyMail dataset, and is powered by the STAM (Stable Training with Adaptive Momentum) optimizer.

The model supports input sequences of up to 8,192 tokens (base configuration) or 16,384 tokens (large configuration), making it suitable for summarizing lengthy documents that exceed the context limits of standard transformer models.

Developer

• —Developer: Assem Sabry
• —GitHub: assemsabry/Research-News-AI-Summarizer
• —HuggingFace Model: assemsabry/Research-News-AI-Summarizer
• —PyPI (STAM Optimizer): stam-optimizer
• —STAM GitHub: github.com/assemsabry/stam

STAM Optimizer

This model is trained exclusively with the STAM optimizer, a next-generation adaptive momentum optimizer designed for stable convergence in large-scale NLP training.

[image]

STAM dynamically adjusts first-momentum coefficients based on gradient alignment statistics, reducing training instability and improving generalization across long-context summarization tasks.

Key STAM Hyperparameters:

References:

• —Paper: Stable Training with Adaptive Momentum
• —Author: Assem Sabry
• —Repository: https://github.com/assemsabry/stam
• —Package: https://pypi.org/project/stam-optimizer/
• —Research: https://tokenai.cloud/research/stam

Model Specifications

Training Details

Hardware

• —GPUs: 2x NVIDIA Tesla T4
• —Total VRAM: 32 GB (16 GB per GPU)
• —Environment: CUDA 12.1, PyTorch 2.0+

Training Regime

• —Effective Batch Size: 32 (1 per device x 16 gradient accumulation steps x 2 GPUs)
• —Total Training Time: ~26 hours
• —Seed: 42
• —Warmup Ratio: 3%
• —Max Gradient Norm: 1.0
• —Early Stopping Patience: 2 evaluations

Training Stages (Curriculum Learning)

Training History

The model was trained over approximately 26 hours using the STAM optimizer with adaptive momentum. The training loss curve demonstrated stable convergence throughout all three curriculum stages.

Loss Progression:

Validation ROUGE-Lsum Progression:

Training Throughput:

• —Average step time: ~27 seconds
• —Peak GPU memory usage: ~14.9 GB per GPU
• —Total tokens processed: ~898M (input + target)

Dataset Details

• —Source: abisee/cnn_dailymail
• —Versions Used: 1.0.0, 2.0.0, 3.0.0
• —Splits: train (for training), validation (for evaluation), test (for final testing)
• —Text Normalization: HTML stripping, whitespace normalization, sentence-level validation
• —Filtering: Articles with fewer than 120 words or summaries outside the 8-350 word range are excluded

Data Preprocessing

# Example preprocessing pipeline
from transformers import LEDTokenizer

tokenizer = LEDTokenizer.from_pretrained("assemsabry/Research-News-AI-Summarizer")

# Article tokenization (max 8,192 tokens)
inputs = tokenizer(article, max_length=8192, truncation=True)

# Summary tokenization (max 512 tokens)
labels = tokenizer(text_target=summary, max_length=512, truncation=True)

Evaluation Results

Test Set Performance (CNN/DailyMail v3.0.0, 500 samples)

Performance by Summary Length Bucket

Usage

Quick Start

from transformers import LEDForConditionalGeneration, LEDTokenizer
import torch

model = LEDForConditionalGeneration.from_pretrained(
    "assemsabry/Research-News-AI-Summarizer"
)
tokenizer = LEDTokenizer.from_pretrained(
    "assemsabry/Research-News-AI-Summarizer"
)

article = """
Your long article text here. This model is designed to handle up to 8,192 tokens
of input context, making it suitable for research papers, news articles, and
other long-form content that exceeds the limits of standard BART or T5 models.
"""

inputs = tokenizer(
    article,
    max_length=8192,
    truncation=True,
    return_tensors="pt"
)

# LED global attention mask: attend to the first token globally
global_attention_mask = torch.zeros_like(inputs["input_ids"])
global_attention_mask[:, 0] = 1

summary_ids = model.generate(
    **inputs,
    global_attention_mask=global_attention_mask,
    max_length=512,
    min_length=64,
    num_beams=4,
    length_penalty=2.0,
    no_repeat_ngram_size=3,
    early_stopping=True,
)

summary = tokenizer.decode(summary_ids[0], skip_special_tokens=True)
print(summary)

Batch Inference

articles = [article1, article2, article3]
inputs = tokenizer(
    articles,
    max_length=8192,
    truncation=True,
    padding=True,
    return_tensors="pt"
)
global_attention_mask = torch.zeros_like(inputs["input_ids"])
global_attention_mask[:, 0] = 1

summary_ids = model.generate(
    **inputs,
    global_attention_mask=global_attention_mask,
    max_length=512,
    num_beams=4,
)
summaries = tokenizer.batch_decode(summary_ids, skip_special_tokens=True)

Repository Structure

.
├── src/
│   ├── config.py          # Training configuration with large sequence settings
│   ├── dataset.py         # CNN/DailyMail dataset processing
│   ├── metrics.py         # ROUGE and BERTScore evaluation
│   ├── model.py           # LED model loading and setup
│   ├── optimizer.py       # STAM and STAMLite PyTorch optimizers
│   └── trainer.py         # Custom Seq2Seq trainer with STAM
├── scripts/
│   ├── train.py           # Main multi-stage training script
│   └── evaluate.py        # Evaluation and inference script
├── configs/
│   ├── base_config.yaml   # Base training configuration
│   └── large_config.yaml  # Large-scale training configuration
├── tests/
│   └── test_model.py      # Unit tests for components
├── media/
│   └── stam.png           # STAM optimizer diagram
├── requirements.txt       # Python dependencies
├── setup.py              # Package installation
└── README.md             # This file

Installation

git clone https://github.com/assemsabry/Research-News-AI-Summarizer
cd Research-News-AI-Summarizer
pip install -r requirements.txt

Training

Set your Hugging Face token as an environment variable:

export HF_TOKEN="your_token_here"

Run multi-stage training:

python scripts/train.py --output-dir ./outputs --stages 1 2 3

Run specific stages only:

python scripts/train.py --stages 2 3

Skip Hub upload (local training only):

python scripts/train.py --skip-upload

Training with Custom Config

from src.config import Config
from src.model import load_model_and_tokenizer, setup_system
from src.trainer import build_training_args, build_trainer

config = Config()
config.data.max_input_length = 16384
config.data.max_target_length = 768
config.training.gradient_accumulation_steps = 32

setup_system(config)
model, tokenizer = load_model_and_tokenizer(config)

Evaluation

Evaluate a trained checkpoint:

python scripts/evaluate.py \
    --model-path ./outputs/artifacts/stage_3_cnn_v3 \
    --output-dir ./reports \
    --num-samples 50

The evaluation script produces:

• —human_evaluation.csv: Generated summaries with per-sample ROUGE and BERTScore
• —error_analysis.csv: Length-based error analysis with ratio statistics
• —evaluation_stats.json: Aggregate metrics by length bucket

Testing

Run unit tests:

python -m unittest tests/test_model.py

Tests cover:

• —Text normalization and HTML stripping
• —Dataset validation (min/max word counts)
• —STAM and STAMLite optimizer initialization and step logic
• —Configuration defaults and effective batch size computation

Limitations and Biases

• —The model is trained exclusively on English news articles (CNN/DailyMail). Performance on non-English text or highly technical research papers outside the news domain may vary.
• —Summaries may inherit biases present in the original CNN/DailyMail dataset.
• —The model does not fact-check generated content. Hallucinations can occur on out-of-domain inputs.
• —Maximum input length is 8,192 tokens (base) or 16,384 tokens (large). Documents exceeding this length are truncated from the end.

License

Apache 2.0

Citation

If you use this model in your research, please cite:

@misc{research-news-ai-summarizer,
  title={Research & News AI Summarizer: Fine-tuned LED with STAM Optimizer},
  author={Sabry, Assem},
  year={2025},
  howpublished={\url{https://huggingface.co/assemsabry/Research-News-AI-Summarizer}}
}

Acknowledgments

• —Base model: allenai/led-base-16384 by AllenAI
• —Dataset: CNN/DailyMail
• —Optimizer: STAM by Assem Sabry