noamaanMulla-03/transformers-issues

HuggingFace Transformers GitHub Issues Dataset

Dataset Description

This dataset contains all issues and pull requests (open and closed) from the `huggingface/transformers` GitHub repository, along with their comment threads. It was collected on July 19-20, 2026 via the GitHub REST API and follows the workflow described in the Hugging Face NLP course — Creating your own dataset.

• —Repository: huggingface/transformers
• —Total rows: 41,618 (issues + pull requests)
• —Date range: November 3, 2018 → July 19, 2026
• —Format: Parquet (single shard, train split)

Dataset Structure

Splits

Composition

Key Columns

Top Labels

Collection Methodology

The dataset was collected using a custom Python script (`prepare_dataset.py`) that:

1. 1.Fetched all issues + PRs using the GitHub REST API with since-based pagination (sorted by updated_at ascending) to bypass GitHub's 10,000-record page-number cap. Issues were written incrementally to a JSONL file for crash recovery.
2. 2.Added an `is_pull_request` flag based on whether the pull_request field is present.
3. 3.Pushed the issues-only dataset to the Hub (as a safety checkpoint before the long comment fetch).
4. 4.Fetched comment bodies for every issue with comments > 0 (4,441 zero-comment issues were skipped as an optimization). Comments were fetched with per_page=100 to minimize pagination requests.
5. 5.Pushed the final dataset (issues + comments) to the Hub.

GitHub's authenticated rate limit (5,000 requests/hour) was respected throughout — the script sleeps until the rate-limit window resets when the budget is exhausted, with a one-hour fallback sleep if the reset header is missing.

Known Limitations

• —5,799 missing issue numbers in the range 1–47,417: these correspond to deleted issues/PRs on GitHub that cannot be retrieved via the API. This is expected and unavoidable.
• —65 minor mismatches between the comments count field and the length of comments_text: these occur when comments were added or deleted between the issue fetch and the comment fetch. The comments field reflects the count at fetch time; comments_text reflects the actual comments retrieved.
• —`milestone`, `performed_via_github_app`, `pinned_comment` are 100% null — these fields are not used on this repository.
• —`body` has 251 nulls and 688 empty strings — some issues/PRs simply have no body text.
• —The dataset includes both issues and pull requests (GitHub's issues endpoint returns both). Use the is_pull_request column to filter.

Potential Use Cases

• —Issue classification: Train a model to categorize issues (bug, feature request, question) based on title + body.
• —Bug triage: Predict which label an issue should have, or whether it's likely to be closed.
• —Duplicate detection: Find semantically similar issues using embeddings of title + body.
• —Pull request summarization: Summarize long PR bodies or comment threads.
• —Software engineering research: Analyze issue lifecycle, response times, contributor activity, label distributions, etc.
• —LLM fine-tuning: Build instruction-tuning datasets for "classify this issue" or "summarize this thread" tasks.

Licensing

The dataset is derived from public GitHub issue data from the `huggingface/transformers` repository, which is licensed under Apache 2.0. The dataset is provided under the same license.

Citation

If you use this dataset, please cite:

@misc{transformers-issues-dataset,
  author       = {Noamaan Mulla},
  title        = {HuggingFace Transformers GitHub Issues Dataset},
  year         = {2026},
  url          = {https://huggingface.co/datasets/noamaanMulla-03/transformers-issues},
  note         = {Collected via the GitHub REST API on July 19-20, 2026}
}

Also cite the original repository:

@misc{wolf2019huggingface,
  author       = {Thomas Wolf and Lysandre Debut and Victor Sanh and Julien Chaumond and Clement Delangue and Anthony Moi and Pierric Cistac and Tim Rault and Rémi Louf and Morgan Funtowicz and Joe Davison and Sam Shleifer and Patrick von Platen and Clara Ma and Yacine Jernite and Julien Plu and Canwen Xu and Teven Le Scao and Sylvain Gugger and Mariama Drame and Quentin Lhoest and Alexander M. Rush},
  title        = {HuggingFace's Transformers: State-of-the-art Natural Language Processing},
  year         = {2019},
  publisher    = {GitHub},
  howpublished = {\url{https://github.com/huggingface/transformers}}
}