Mindie/Summary_article_KSS_style_dataset

KSS-Style Summarization Dataset

Dataset Description

This dataset consists of structured summaries generated from news articles.

Each sample contains an original article and a corresponding model-generated summary in a structured format. The dataset is designed to provide high-quality, format-consistent summaries suitable for training and evaluation of summarization models.

Data Source

• —Original data: CNN news articles (original_article)

Data Generation Pipeline

The dataset was created through a multi-stage generation and filtering pipeline:

1. 1.Initial Generation Summaries were generated using a base language model with a system prompt enforcing a structured output format.

1. 1.Algorithmic Filtering Generated outputs were validated using rule-based methods to ensure:

• —Presence of required fields (Subject, Keywords, Summary)
• —Proper formatting
• —Reasonable summary length
• —Minimum number of keywords

1. 1.Regeneration Samples that failed validation were re-generated to improve structural consistency.

1. 1.Semantic Similarity Evaluation The similarity between the original article and the generated summary was computed using an embedding-based model.

1. 1.Sampling Strategy Data was sampled from different similarity ranges:

• —High similarity
• —Low similarity
• —Random samples (each approximately 10%)

1. 1.Quality Evaluation A separate language model was used to evaluate the generated summaries based on:

• —Hallucination
• —Contradiction
• —Relevance

Each sample was assigned a reliability score (0–100).

1. 1.Final Selection Samples were reviewed and selected based on evaluation scores and included in the final dataset.

Output Format

Each generated_output follows the structured format:

Subject: ... Keywords: ... Summary: ...

Dataset Structure

Each sample contains:

• —original_article: Source news article
• —model_output: Structured summary
• —subject: Extracted subject from the generated output
• —keywords: Extracted keywords
• —summary: Extracted summary

Evaluation

During dataset construction, generated summaries were evaluated using two types of signals:

• —Semantic Similarity Measures alignment between the original article and the generated summary.

• —Reliability Evaluation Assesses factual consistency and overall quality using a language model.

The evaluation considers:

• —Hallucination (unsupported information)
• —Contradiction (conflicts with source)
• —Relevance (topic alignment)

Design Rationale

The dataset focuses on structured summaries (KSS-style) to enforce consistency in output format.

For evaluation efficiency, similarity computations were performed on the summary only. This is based on the autoregressive nature of language models, where all generated fields (Subject, Keywords, Summary) are produced under a shared sequential context. As a result, the summary—being the most information-dense component—serves as a strong proxy for overall output quality.

Limitations

• —The dataset is generated using language models and may contain minor inconsistencies.
• —Evaluation scores are model-based and may not fully reflect human judgment.
• —Structured format constraints may limit stylistic diversity.

Repository

The full data generation and filtering pipeline is available at:

[GitHub Repository Link Here]

Notes

• —Only structured (KSS-style) summaries are included in the final dataset.
• —Intermediate data (e.g., unstructured summaries) is not included but can be reproduced using the provided pipeline.
• —used_model: Qwen3-4b(generation for structured summary), google/embeddinggemma-300m(similarity between original article and summary), google/gemma-3-4b-it(Evaluation for generated data's reliability)

dataset_info: features:

• —name: original_article dtype: string
• —name: model_output dtype: string
• —name: subject dtype: string
• —name: keywords dtype: string
• —name: summary dtype: string splits:
• —name: train numbytes: 5915830 numexamples: 961 downloadsize: 3612388 datasetsize: 5915830 configs:
• —configname: default datafiles:
• —split: train path: data/train-* ---