devleoespinosa/qus-user-story-quality-refined

Refined QUS User Story Quality Dataset

Dataset Summary

This repository contains a refined and relabeled adaptation of the user-story dataset reported by Sharma and Tripathi (2025) for the evaluation of user story quality according to the Quality User Story (QUS) framework.

The original corpus contains 960 criterion-specific user-story instances organized around the eight individual quality criteria of QUS. During the preparation of the experiments reported in our study, inconsistencies were identified in the original labeling. The corpus was therefore manually reviewed and relabeled using the formal definitions of the QUS criteria while preserving the original set of 960 criterion-specific instances.

This refined version is the dataset used for the experiments presented in:

Fine-Tuned DistilBERT for Automated User Story Quality Assessment Leonardo Espinosa Arévalo, Antonio Armando Aguileta Güemez, and Raúl Antonio Aguilar Vera. 2026. Preprint manuscript.

The definitive Preprints.org DOI and citation will be added once the preprint is published.

Relationship to the Original Dataset

This dataset is not an independently created corpus.

It is derived from the dataset reported by:

Sharma, A., & Tripathi, A. K. (2025). Evaluating user story quality with LLMs: a comparative study. Journal of Intelligent Information Systems, 63, 1423–1451. DOI: 10.1007/s10844-025-00939-3

The original dataset was used by Sharma and Tripathi to evaluate user story quality using generative large language models under the individual criteria of the Quality User Story framework.

Our study reuses the same set of 960 criterion-specific instances to enable a controlled comparison with specialized encoder-based Transformer models.

However, before training and evaluating the models, the labels were reviewed because inconsistencies were detected in the original annotations.

The dataset distributed in this repository corresponds to the refined labeling used in our experiments, not to the original labeling reported by Sharma and Tripathi.

Refinement and Relabeling Process

During preliminary analysis of the source dataset, cases were identified in which identical user stories appeared within the same QUS criterion with contradictory labels.

For example, some instances occurred both as compliant and non-compliant with the same quality criterion.

To address these inconsistencies, the dataset was manually reviewed using the formal definitions of the eight individual QUS criteria.

The refinement process followed these principles:

• —The original set of 960 criterion-specific instances was preserved.
• —Each individual QUS criterion was evaluated independently.
• —Existing labels were reviewed for semantic consistency.
• —Contradictory annotations were resolved.
• —Repeated instances were preserved when they belonged to the original corpus.
• —Identical story/criterion pairs no longer retain contradictory labels after refinement.
• —No artificial oversampling, undersampling, or other class-balancing technique was applied.

Consequently, the dataset retains the experimental structure of the source corpus while providing the corrected ground-truth labels used in our study.

Use in the Associated Study

This dataset constitutes the experimental corpus used in:

Fine-Tuned DistilBERT for Automated User Story Quality Assessment

The study investigates whether specialized encoder-based Transformer models can automatically detect violations of individual QUS quality criteria in user stories.

Three encoder architectures were experimentally evaluated:

• —BERT
• —RoBERTa
• —DistilBERT

An independent binary classification task was defined for each of the eight individual QUS criteria.

The DistilBERT models resulting from the study are publicly distributed through the associated Hugging Face Collection.

The dataset in this repository therefore represents the exact refined dataset on which the training, cross-validation, evaluation, and statistical analyses reported in the study are based.

QUS Criteria

The dataset covers the eight quality criteria of the QUS framework that can be evaluated independently for a single user story:

1. 1.Atomic
2. 2.Conceptually Sound
3. 3.Estimable
4. 4.Full Sentence
5. 5.Minimal
6. 6.Problem Oriented
7. 7.Unambiguous
8. 8.Well Formed

Each criterion contains exactly 120 criterion-specific instances, resulting in a total of 960 instances.

The five collective QUS criteria that require analysis across multiple user stories are outside the scope of this dataset.

Dataset Structure

Each row represents the evaluation of a user story against one specific QUS criterion.

Label Semantics

• —0 — Incorrect: the user story violates the evaluated QUS criterion.
• —1 — Correct: the user story complies with the evaluated QUS criterion.

The label should therefore be interpreted only with respect to the criterion specified in the same row.

A label of correct does not imply that the user story satisfies every QUS criterion.

Class Distribution

The following distribution corresponds to the refined labels used in the associated study.

No artificial class-balancing procedure was applied.

Dataset Split

The repository exposes a single split:

full

No fixed training, validation, or test partition is provided.

In the associated study, each QUS criterion was evaluated independently using five-fold cross-validation. Therefore, publishing an arbitrary fixed train/test partition would not reproduce the experimental protocol used in the paper.

Researchers wishing to reproduce the reported experiments should construct the folds from the complete dataset according to the methodology described in the associated preprint.

Associated DistilBERT Models

Eight fine-tuned DistilBERT classifiers were produced from this experimental framework, one for each individual QUS criterion:

• —devleoespinosa/DistilBERT-AUSQ-SL-Atomic
• —devleoespinosa/DistilBERT-AUSQ-SL-Conceptually-Sound
• —devleoespinosa/DistilBERT-AUSQ-SL-Estimable
• —devleoespinosa/DistilBERT-AUSQ-SL-Full-Sentence
• —devleoespinosa/DistilBERT-AUSQ-SL-Minimal
• —devleoespinosa/DistilBERT-AUSQ-SL-Problem-Oriented
• —devleoespinosa/DistilBERT-AUSQ-SL-Unambiguous
• —devleoespinosa/DistilBERT-AUSQ-SL-Well-Formed

The models and this dataset are grouped in the Hugging Face Collection associated with the study.

Intended Uses

The dataset is intended primarily for research on:

• —automated user story quality assessment,
• —requirements engineering,
• —natural language processing for software engineering,
• —binary text classification,
• —QUS quality-criterion detection,
• —evaluation of encoder-based language models,
• —comparison of specialized and general-purpose language models.

It may also serve as a benchmark for future approaches to automated requirements-quality assessment.

Limitations

This dataset should be interpreted considering several limitations.

First, the user stories originate from a synthetic corpus rather than from a representative sample of requirements collected directly from industrial software projects.

Second, each instance evaluates only one QUS criterion at a time. The binary label therefore represents criterion-specific compliance rather than overall user story quality.

Third, the dataset covers only the eight individual criteria of the QUS framework. Collective quality criteria requiring relationships among multiple user stories are not represented.

Finally, although the labels were manually reviewed to resolve inconsistencies in the source dataset, quality assessment in requirements engineering may still involve judgment in cases where natural-language requirements admit multiple reasonable interpretations.

Dataset Provenance

The provenance of this release can be summarized as:

Sharma & Tripathi (2025) Original 960 criterion-specific user-story instances and original labels

↓

Manual review and relabeling Identification and correction of semantic labeling inconsistencies according to the QUS definitions

↓

Refined QUS User Story Quality Dataset 960 instances with the corrected labels distributed in this repository

↓

Fine-Tuned DistilBERT for Automated User Story Quality Assessment (2026) Dataset used for model fine-tuning, five-fold cross-validation, evaluation, and statistical comparison

This distinction between the source corpus and the refined experimental dataset is important for the reproducibility and interpretation of the associated results.

Associated Publication

This dataset is released as supporting research material for:

Espinosa Arévalo, L., Aguileta Güemez, A. A., & Aguilar Vera, R. A. Fine-Tuned DistilBERT for Automated User Story Quality Assessment. 2026.

The work evaluates specialized BERT, RoBERTa, and DistilBERT classifiers against generative-model baselines for detecting violations of individual QUS criteria.

The definitive citation and DOI will be added after publication on Preprints.org.

Original Dataset Reference

Sharma, A., & Tripathi, A. K. (2025).

Evaluating user story quality with LLMs: a comparative study.

Journal of Intelligent Information Systems, 63, 1423–1451.

DOI: 10.1007/s10844-025-00939-3

Authors of the Refined Dataset

IS. Leonardo Espinosa Arévalo

ORCID: 0009-0006-0932-5751

Dr. Antonio Armando Aguileta Güemez

ORCID: 0000-0001-5155-3543

Dr. Raúl Antonio Aguilar Vera

ORCID: 0000-0002-1711-7016

Citation

Until the associated preprint receives its DOI, please cite both the refined dataset and the source study.