open-index/open-github-issues

OpenGitHub Issues

What is it?

The full development metadata of 7 public GitHub repositories, fetched from the GitHub REST API and GraphQL API, converted to Parquet and hosted here for easy access.

Right now the archive has 6.0M rows across 8 tables in 699.6 MB of Zstd-compressed Parquet. Every issue, pull request, comment, code review, timeline event, file change, and CI status check is stored as a separate table you can load individually or query together.

This is the companion to OpenGitHub, which mirrors the real-time GitHub event stream via GH Archive. That dataset tells you what happened across all of GitHub. This one gives you the full picture for specific repos: complete issue threads, full PR review conversations, the state machine from open to close.

People use it for:

• —Code review research with inline comments attached to specific diff lines
• —Project health metrics like merge rates, review turnaround, label usage
• —Issue triage and classification with full text, labels, and timeline
• —Software engineering process mining from timeline event sequences

Last updated: 2026-06-06 23:37 UTC.

Latest Sync

New items since the previous publish:

Repositories

How to download and use this dataset

Data lives at data/{table}/{owner}/{repo}/0.parquet. Load a single table, a single repo, or everything at once. Standard Hugging Face Parquet layout, works with DuckDB, datasets, pandas, and huggingface_hub out of the box.

Using DuckDB

DuckDB reads Parquet directly from Hugging Face, no download step needed. Save any query below as a .sql file and run it with duckdb < query.sql.

-- Top issue authors across all repos
SELECT
    author,
    COUNT(*) as issue_count,
    COUNT(*) FILTER (WHERE state = 'open') as open,
    COUNT(*) FILTER (WHERE state = 'closed') as closed
FROM read_parquet('hf://datasets/open-index/open-github-issues/data/issues/**/0.parquet')
WHERE is_pull_request = false
GROUP BY author
ORDER BY issue_count DESC
LIMIT 20;

-- PR merge rate by repo
SELECT
    split_part(filename, '/', 8) || '/' || split_part(filename, '/', 9) as repo,
    COUNT(*) as total_prs,
    COUNT(*) FILTER (WHERE merged) as merged,
    ROUND(COUNT(*) FILTER (WHERE merged) * 100.0 / COUNT(*), 1) as merge_pct
FROM read_parquet('hf://datasets/open-index/open-github-issues/data/pull_requests/**/0.parquet', filename=true)
GROUP BY repo
ORDER BY total_prs DESC;

-- Most reviewed PRs by number of review submissions
SELECT
    r.pr_number,
    COUNT(*) as review_count,
    COUNT(*) FILTER (WHERE r.state = 'APPROVED') as approvals,
    COUNT(*) FILTER (WHERE r.state = 'CHANGES_REQUESTED') as changes_requested
FROM read_parquet('hf://datasets/open-index/open-github-issues/data/reviews/**/0.parquet') r
GROUP BY r.pr_number
ORDER BY review_count DESC
LIMIT 20;

-- Label activity over time (monthly)
SELECT
    date_trunc('month', created_at) as month,
    COUNT(*) as label_events
FROM read_parquet('hf://datasets/open-index/open-github-issues/data/timeline_events/**/0.parquet')
WHERE event_type = 'LabeledEvent'
GROUP BY month
ORDER BY month;

-- Largest PRs by lines changed
SELECT
    number,
    additions,
    deletions,
    changed_files,
    additions + deletions as total_lines
FROM read_parquet('hf://datasets/open-index/open-github-issues/data/pull_requests/**/0.parquet')
ORDER BY total_lines DESC
LIMIT 20;

Using Python (uv run)

These scripts use PEP 723 inline metadata. Save as a .py file and run with uv run script.py. No virtualenv or pip install needed.

Stream issues:

# /// script
# requires-python = ">=3.11"
# dependencies = ["datasets"]
# ///
from datasets import load_dataset

ds = load_dataset("open-index/open-github-issues", "issues", streaming=True)
for i, row in enumerate(ds["train"]):
    print(f"#{row['number']}: [{row['state']}] {row['title']} (by {row['author']})")
    if i >= 19:
        break

Load a specific repo:

# /// script
# requires-python = ">=3.11"
# dependencies = ["datasets"]
# ///
from datasets import load_dataset

ds = load_dataset(
    "open-index/open-github-issues",
    "pull_requests",
    data_files="data/pull_requests/facebook/react/0.parquet",
)
df = ds["train"].to_pandas()
print(f"Loaded {len(df)} pull requests")
print(f"Merged: {df['merged'].sum()} ({df['merged'].mean()*100:.1f}%)")
print(f"\nTop 10 by lines changed:")
df["total_lines"] = df["additions"] + df["deletions"]
print(df.nlargest(10, "total_lines")[["number", "additions", "deletions", "total_lines"]].to_string(index=False))

Download files:

# /// script
# requires-python = ">=3.11"
# dependencies = ["huggingface-hub"]
# ///
from huggingface_hub import snapshot_download

# Download only issues
snapshot_download(
    "open-index/open-github-issues",
    repo_type="dataset",
    local_dir="./open-github-issues/",
    allow_patterns="data/issues/**/*.parquet",
)
print("Downloaded issues parquet files to ./open-github-issues/")

For faster downloads, install pip install huggingface_hub[hf_transfer] and set HF_HUB_ENABLE_HF_TRANSFER=1.

Dataset structure

issues

Both issues and PRs live in this table (check is_pull_request). Join with pull_requests on number for PR-specific fields like merge status and diff stats.

pull_requests

PR-specific fields. Join with issues on number for title, body, labels, and other shared fields.

comments

Conversation comments on issues and PRs. These are the threaded discussion comments, not inline code review comments (those are in review_comments).

review_comments

Inline code review comments on PR diffs. Each comment is attached to a specific file and line in the diff.

reviews

PR review decisions. One row per review action on a PR.

timeline_events

The full lifecycle of every issue and PR. Every label change, assignment, cross-reference, merge, force-push, lock, and other state transition.

Event types: labeled, unlabeled, closed, reopened, assigned, unassigned, milestoned, demilestoned, renamed, cross-referenced, referenced, locked, unlocked, pinned, merged, review_requested, head_ref_force_pushed, head_ref_deleted, ready_for_review, convert_to_draft, and more.

Common fields (actor, created_at, database_id and extracted columns above) are stored in dedicated columns and removed from data to reduce storage. The data field contains only remaining event-specific payload. See the GitHub GraphQL timeline items documentation for the full type catalog.

pr_files

Every file touched by each pull request, with per-file diff statistics.

commit_statuses

CI/CD status checks and GitHub Actions results for each commit.

Dataset statistics

How it's built

The sync pipeline uses both GitHub APIs. The REST API handles bulk listing: issues, comments, and review comments are fetched repo-wide with since-based incremental pagination and parallel page fetching across multiple tokens. The GraphQL API handles per-item detail: one query grabs reviews, timeline events, file changes, and commit statuses in a single round trip, with automatic REST fallback for PRs with more than 100 files or reviews.

Multiple GitHub Personal Access Tokens rotate round-robin to spread rate limit load. The pipeline is fully incremental and idempotent: re-running picks up only what changed since the last sync.

Everything lands in per-repo DuckDB files first, then gets exported to Parquet with Zstd compression for publishing here. No filtering, deduplication, or content changes. Bot activity, automated PRs, CI noise, Dependabot upgrades, all of it is preserved, because that's how repos actually work.

Known limitations

• —Point-in-time snapshot. Data reflects the state at the last sync, not real-time. Incremental updates capture everything that changed since the previous sync.
• —Bot activity included. Comments and PRs from bots (Dependabot, Renovate, GitHub Actions, etc.) are included without filtering. This is intentional. Filter on author if you want humans only.
• —JSON columns. labels, assignees, reactions, and data contain JSON strings. Use json_extract() in DuckDB or json.loads() in Python.
• —Body text can be large. Issue and comment bodies contain full Markdown, sometimes with embedded images. Project only the columns you need for memory-constrained workloads.
• —Timeline data varies by event type. The data field in timeline_events contains the raw event payload as JSON. The schema depends on event_type.

Personal and sensitive information

Usernames, user IDs, and author associations are included as they appear in the GitHub API. All data was already publicly accessible on GitHub. Email addresses do not appear in this dataset (they exist only in git commit objects, which are in the separate code archive, not here). No private repository data is present.

License

Released under the Open Data Commons Attribution License (ODC-By) v1.0. The underlying data is sourced from GitHub's public API. GitHub's Terms of Service apply to the original data.

Thanks

All the data here comes from GitHub's public REST API and GraphQL API. We are grateful to the open-source maintainers and contributors whose work is represented in these tables.

• —[OpenGitHub](https://huggingface.co/datasets/open-index/open-github), our companion dataset covering the full GitHub event stream via GH Archive by Ilya Grigorik
• —Built with DuckDB (Go driver), Apache Parquet (Zstd compression), published via Hugging Face Hub

Questions, feedback, or issues? Open a discussion on the Community tab.