open-index/open-github-issues

1---2license: odc-by3task_categories:4- feature-extraction5language:6- en7- mul8pretty_name: OpenGitHub Issues9size_categories:10- 1M<n<10M11tags:12- github13- metadata14- issues15- pull-requests16- code-review17- open-source18- software-engineering19configs:20- config_name: issues21  data_files: "data/issues/**/*.parquet"22- config_name: pull_requests23  data_files: "data/pull_requests/**/*.parquet"24- config_name: comments25  data_files: "data/comments/**/*.parquet"26- config_name: review_comments27  data_files: "data/review_comments/**/*.parquet"28- config_name: reviews29  data_files: "data/reviews/**/*.parquet"30- config_name: timeline_events31  data_files: "data/timeline_events/**/*.parquet"32- config_name: pr_files33  data_files: "data/pr_files/**/*.parquet"34- config_name: commit_statuses35  data_files: "data/commit_statuses/**/*.parquet"36---37 38# OpenGitHub Issues39 40## What is it?41 42The full development metadata of 7 public GitHub repositories, fetched from the [GitHub REST API](https://docs.github.com/en/rest) and [GraphQL API](https://docs.github.com/en/graphql), converted to Parquet and hosted here for easy access.43 44Right now the archive has **6.0M rows** across 8 tables in **699.6 MB** of Zstd-compressed Parquet. Every issue, pull request, comment, code review, timeline event, file change, and CI status check is stored as a separate table you can load individually or query together.45 46This is the companion to [OpenGitHub](https://huggingface.co/datasets/open-index/open-github), which mirrors the real-time GitHub event stream via [GH Archive](https://www.gharchive.org/). That dataset tells you what happened across all of GitHub. This one gives you the full picture for specific repos: complete issue threads, full PR review conversations, the state machine from open to close.47 48People use it for:49 50- **Code review research** with inline comments attached to specific diff lines51- **Project health metrics** like merge rates, review turnaround, label usage52- **Issue triage and classification** with full text, labels, and timeline53- **Software engineering process mining** from timeline event sequences54 55Last updated: **2026-06-06 23:37 UTC**.56 57 58## Latest Sync59 60New items since the previous publish:61 62| Repository | Issues | PRs | Comments | Reviews | Timeline |63|---|---:|---:|---:|---:|---:|64| **golang/go** | +76.8K | +5.2K | — | +305 | +768.4K |65| **mdn/content** | +42.2K | +31.9K | +81.0K | +82.4K | +648.7K |66| **python/cpython** | +148.1K | +71.8K | — | +17.9K | +261.6K |67| **rust-lang/rust** | +156.4K | +93.9K | — | — | +16.6K |68| **swiftlang/swift** | +85.7K | +67.9K | — | +110.8K | +1.6M |69| **vuejs/core** | +12.3K | +6.2K | +36.5K | +5.0K | +130.1K |70 71 72## Repositories73 74| Repository | Issues | PRs | Comments | Reviews | Timeline | Total | Last Updated |75|---|---:|---:|---:|---:|---:|---:|---|76| **golang/go** | 76.8K | 5.2K | 0 | 305 | 768.4K | 927.8K | 2026-06-04 06:54 UTC |77| **mdn/content** | 42.2K | 31.9K | 81.0K | 82.4K | 648.7K | 1.3M | 2026-06-02 09:13 UTC |78| **python/cpython** | 148.1K | 71.8K | 0 | 17.9K | 261.6K | 608.5K | 2026-06-02 04:39 UTC |79| **rust-lang/rust** | 156.4K | 93.9K | 0 | 0 | 16.6K | 275.3K | 2026-06-02 23:53 UTC |80| **swiftlang/swift** | 85.7K | 67.9K | 0 | 110.8K | 1.6M | 2.6M | 2026-06-03 01:26 UTC |81| **vuejs/core** | 12.3K | 6.2K | 36.5K | 5.0K | 130.1K | 227.9K | 2026-06-02 19:28 UTC |82| **vuejs/docs** | 3.3K | 2.3K | 7.1K | 2.7K | 35.4K | 66.7K | 2026-06-02 09:26 UTC |83 84## How to download and use this dataset85 86Data lives at `data/{table}/{owner}/{repo}/0.parquet`. Load a single table, a single repo, or everything at once. Standard Hugging Face Parquet layout, works with DuckDB, `datasets`, `pandas`, and `huggingface_hub` out of the box.87 88### Using DuckDB89 90DuckDB reads Parquet directly from Hugging Face, no download step needed. Save any query below as a `.sql` file and run it with `duckdb < query.sql`.91 92```sql93-- Top issue authors across all repos94SELECT95    author,96    COUNT(*) as issue_count,97    COUNT(*) FILTER (WHERE state = 'open') as open,98    COUNT(*) FILTER (WHERE state = 'closed') as closed99FROM read_parquet('hf://datasets/open-index/open-github-issues/data/issues/**/0.parquet')100WHERE is_pull_request = false101GROUP BY author102ORDER BY issue_count DESC103LIMIT 20;104```105 106```sql107-- PR merge rate by repo108SELECT109    split_part(filename, '/', 8) || '/' || split_part(filename, '/', 9) as repo,110    COUNT(*) as total_prs,111    COUNT(*) FILTER (WHERE merged) as merged,112    ROUND(COUNT(*) FILTER (WHERE merged) * 100.0 / COUNT(*), 1) as merge_pct113FROM read_parquet('hf://datasets/open-index/open-github-issues/data/pull_requests/**/0.parquet', filename=true)114GROUP BY repo115ORDER BY total_prs DESC;116```117 118```sql119-- Most reviewed PRs by number of review submissions120SELECT121    r.pr_number,122    COUNT(*) as review_count,123    COUNT(*) FILTER (WHERE r.state = 'APPROVED') as approvals,124    COUNT(*) FILTER (WHERE r.state = 'CHANGES_REQUESTED') as changes_requested125FROM read_parquet('hf://datasets/open-index/open-github-issues/data/reviews/**/0.parquet') r126GROUP BY r.pr_number127ORDER BY review_count DESC128LIMIT 20;129```130 131```sql132-- Label activity over time (monthly)133SELECT134    date_trunc('month', created_at) as month,135    COUNT(*) as label_events136FROM read_parquet('hf://datasets/open-index/open-github-issues/data/timeline_events/**/0.parquet')137WHERE event_type = 'LabeledEvent'138GROUP BY month139ORDER BY month;140```141 142```sql143-- Largest PRs by lines changed144SELECT145    number,146    additions,147    deletions,148    changed_files,149    additions + deletions as total_lines150FROM read_parquet('hf://datasets/open-index/open-github-issues/data/pull_requests/**/0.parquet')151ORDER BY total_lines DESC152LIMIT 20;153```154 155### Using Python (`uv run`)156 157These scripts use [PEP 723](https://peps.python.org/pep-0723/) inline metadata. Save as a `.py` file and run with `uv run script.py`. No virtualenv or `pip install` needed.158 159**Stream issues:**160 161```python162# /// script163# requires-python = ">=3.11"164# dependencies = ["datasets"]165# ///166from datasets import load_dataset167 168ds = load_dataset("open-index/open-github-issues", "issues", streaming=True)169for i, row in enumerate(ds["train"]):170    print(f"#{row['number']}: [{row['state']}] {row['title']} (by {row['author']})")171    if i >= 19:172        break173```174 175**Load a specific repo:**176 177```python178# /// script179# requires-python = ">=3.11"180# dependencies = ["datasets"]181# ///182from datasets import load_dataset183 184ds = load_dataset(185    "open-index/open-github-issues",186    "pull_requests",187    data_files="data/pull_requests/facebook/react/0.parquet",188)189df = ds["train"].to_pandas()190print(f"Loaded {len(df)} pull requests")191print(f"Merged: {df['merged'].sum()} ({df['merged'].mean()*100:.1f}%)")192print(f"\nTop 10 by lines changed:")193df["total_lines"] = df["additions"] + df["deletions"]194print(df.nlargest(10, "total_lines")[["number", "additions", "deletions", "total_lines"]].to_string(index=False))195```196 197**Download files:**198 199```python200# /// script201# requires-python = ">=3.11"202# dependencies = ["huggingface-hub"]203# ///204from huggingface_hub import snapshot_download205 206# Download only issues207snapshot_download(208    "open-index/open-github-issues",209    repo_type="dataset",210    local_dir="./open-github-issues/",211    allow_patterns="data/issues/**/*.parquet",212)213print("Downloaded issues parquet files to ./open-github-issues/")214```215 216For faster downloads, install `pip install huggingface_hub[hf_transfer]` and set `HF_HUB_ENABLE_HF_TRANSFER=1`.217 218## Dataset structure219 220### `issues`221 222Both issues and PRs live in this table (check `is_pull_request`). Join with `pull_requests` on `number` for PR-specific fields like merge status and diff stats.223 224| Column | Type | Description |225|---|---|---|226| `number` | int32 | Issue/PR number (primary key) |227| `node_id` | string | GitHub GraphQL node ID |228| `is_pull_request` | bool | True if this is a PR |229| `title` | string | Title |230| `body` | string | Full body text in Markdown |231| `state` | string | `open` or `closed` |232| `state_reason` | string | `completed`, `not_planned`, or `reopened` |233| `author` | string | Username of the creator |234| `created_at` | timestamp | When opened |235| `updated_at` | timestamp | Last activity |236| `closed_at` | timestamp | When closed (null if open) |237| `labels` | string (JSON) | Array of label names |238| `assignees` | string (JSON) | Array of assignee usernames |239| `milestone_title` | string | Milestone name |240| `milestone_number` | int32 | Milestone number |241| `reactions` | string (JSON) | Reaction counts (`{"+1": 5, "heart": 2}`) |242| `comment_count` | int32 | Number of comments |243| `locked` | bool | Whether the conversation is locked |244| `lock_reason` | string | Lock reason |245 246### `pull_requests`247 248PR-specific fields. Join with `issues` on `number` for title, body, labels, and other shared fields.249 250| Column | Type | Description |251|---|---|---|252| `number` | int32 | PR number (matches `issues.number`) |253| `merged` | bool | Whether the PR was merged |254| `merged_at` | timestamp | When merged |255| `merged_by` | string | Username who merged |256| `merge_commit_sha` | string | Merge commit SHA |257| `base_ref` | string | Target branch (e.g. `main`) |258| `head_ref` | string | Source branch |259| `head_sha` | string | Head commit SHA |260| `additions` | int32 | Lines added |261| `deletions` | int32 | Lines deleted |262| `changed_files` | int32 | Number of files changed |263| `draft` | bool | Whether the PR is a draft |264| `maintainer_can_modify` | bool | Whether maintainers can push to the head branch |265 266### `comments`267 268Conversation comments on issues and PRs. These are the threaded discussion comments, not inline code review comments (those are in `review_comments`).269 270| Column | Type | Description |271|---|---|---|272| `id` | int64 | Comment ID (primary key) |273| `issue_number` | int32 | Parent issue/PR number |274| `author` | string | Username |275| `body` | string | Comment body in Markdown |276| `created_at` | timestamp | When posted |277| `updated_at` | timestamp | Last edit |278| `reactions` | string (JSON) | Reaction counts |279| `author_association` | string | `OWNER`, `MEMBER`, `CONTRIBUTOR`, `NONE`, etc. |280 281### `review_comments`282 283Inline code review comments on PR diffs. Each comment is attached to a specific file and line in the diff.284 285| Column | Type | Description |286|---|---|---|287| `id` | int64 | Comment ID (primary key) |288| `pr_number` | int32 | Parent PR number |289| `review_id` | int64 | Parent review ID |290| `author` | string | Reviewer username |291| `body` | string | Comment body in Markdown |292| `path` | string | File path in the diff |293| `line` | int32 | Line number |294| `side` | string | `LEFT` (old code) or `RIGHT` (new code) |295| `diff_hunk` | string | Surrounding diff context |296| `created_at` | timestamp | When posted |297| `updated_at` | timestamp | Last edit |298| `in_reply_to_id` | int64 | Parent comment ID (for threaded replies) |299 300### `reviews`301 302PR review decisions. One row per review action on a PR.303 304| Column | Type | Description |305|---|---|---|306| `id` | int64 | Review ID (primary key) |307| `pr_number` | int32 | Parent PR number |308| `author` | string | Reviewer username |309| `state` | string | `APPROVED`, `CHANGES_REQUESTED`, `COMMENTED`, `DISMISSED` |310| `body` | string | Review summary in Markdown |311| `submitted_at` | timestamp | When submitted |312| `commit_id` | string | Commit SHA that was reviewed |313 314### `timeline_events`315 316The full lifecycle of every issue and PR. Every label change, assignment, cross-reference, merge, force-push, lock, and other state transition.317 318| Column | Type | Description |319|---|---|---|320| `id` | string | Event ID (node_id or synthesized) |321| `issue_number` | int32 | Parent issue/PR number |322| `event_type` | string | Event type (see below) |323| `actor` | string | Username who triggered the event |324| `created_at` | timestamp | When it happened |325| `database_id` | int64 | GitHub database ID for the event |326| `label_name` | string | Label name (`labeled`, `unlabeled`) |327| `label_color` | string | Label hex color |328| `state_reason` | string | Close reason: `COMPLETED`, `NOT_PLANNED` (`closed`) |329| `assignee_login` | string | Username assigned/unassigned (`assigned`, `unassigned`) |330| `milestone_title` | string | Milestone name (`milestoned`, `demilestoned`) |331| `title_from` | string | Previous title before rename (`renamed`) |332| `title_to` | string | New title after rename (`renamed`) |333| `ref_type` | string | Referenced item type: `Issue` or `PullRequest` (`cross-referenced`, `referenced`) |334| `ref_number` | int32 | Referenced issue/PR number |335| `ref_url` | string | URL of the referenced item |336| `will_close` | bool | Whether the reference will close this issue |337| `lock_reason` | string | Lock reason (`locked`) |338| `data` | string (JSON) | Remaining event-specific payload (common fields stripped) |339 340Event types: `labeled`, `unlabeled`, `closed`, `reopened`, `assigned`, `unassigned`, `milestoned`, `demilestoned`, `renamed`, `cross-referenced`, `referenced`, `locked`, `unlocked`, `pinned`, `merged`, `review_requested`, `head_ref_force_pushed`, `head_ref_deleted`, `ready_for_review`, `convert_to_draft`, and more.341 342Common fields (`actor`, `created_at`, `database_id` and extracted columns above) are stored in dedicated columns and removed from `data` to reduce storage. The `data` field contains only remaining event-specific payload. See the [GitHub GraphQL timeline items documentation](https://docs.github.com/en/graphql/reference/unions#issuetimelineitems) for the full type catalog.343 344### `pr_files`345 346Every file touched by each pull request, with per-file diff statistics.347 348| Column | Type | Description |349|---|---|---|350| `pr_number` | int32 | Parent PR number |351| `path` | string | File path |352| `additions` | int32 | Lines added |353| `deletions` | int32 | Lines deleted |354| `status` | string | `added`, `removed`, `modified`, `renamed` |355| `previous_filename` | string | Original path (for renames) |356 357### `commit_statuses`358 359CI/CD status checks and GitHub Actions results for each commit.360 361| Column | Type | Description |362|---|---|---|363| `sha` | string | Commit SHA |364| `context` | string | Check name (e.g. `ci/circleci`, `check:build`) |365| `state` | string | `success`, `failure`, `pending`, `error` |366| `description` | string | Status description |367| `target_url` | string | Link to CI details |368| `created_at` | timestamp | When reported |369 370## Dataset statistics371 372| Table | Rows | Description |373|-------|-----:|-------------|374| `issues` | 524.7K | Issues and pull requests (shared metadata) |375| `pull_requests` | 279.2K | PR-specific fields (merge status, diffs, refs) |376| `comments` | 112.8K | Conversation comments on issues and PRs |377| `review_comments` | 11.8K | Inline code review comments on PR diffs |378| `reviews` | 219.1K | PR review decisions |379| `timeline_events` | 3.5M | Activity timeline (labels, closes, merges, assignments) |380| `pr_files` | 1.3M | Files changed in each pull request |381| `commit_statuses` | 76.2K | CI/CD status checks per commit |382| **Total** | **6.0M** | |383 384## How it's built385 386The sync pipeline uses both GitHub APIs. The [REST API](https://docs.github.com/en/rest) handles bulk listing: issues, comments, and review comments are fetched repo-wide with `since`-based incremental pagination and parallel page fetching across multiple tokens. The [GraphQL API](https://docs.github.com/en/graphql) handles per-item detail: one query grabs reviews, timeline events, file changes, and commit statuses in a single round trip, with automatic REST fallback for PRs with more than 100 files or reviews.387 388Multiple GitHub Personal Access Tokens rotate round-robin to spread rate limit load. The pipeline is fully incremental and idempotent: re-running picks up only what changed since the last sync.389 390Everything lands in per-repo [DuckDB](https://duckdb.org/) files first, then gets exported to Parquet with Zstd compression for publishing here. No filtering, deduplication, or content changes. Bot activity, automated PRs, CI noise, Dependabot upgrades, all of it is preserved, because that's how repos actually work.391 392## Known limitations393 394- **Point-in-time snapshot.** Data reflects the state at the last sync, not real-time. Incremental updates capture everything that changed since the previous sync.395- **Bot activity included.** Comments and PRs from bots (Dependabot, Renovate, GitHub Actions, etc.) are included without filtering. This is intentional. Filter on `author` if you want humans only.396- **JSON columns.** `labels`, `assignees`, `reactions`, and `data` contain JSON strings. Use `json_extract()` in DuckDB or `json.loads()` in Python.397- **Body text can be large.** Issue and comment bodies contain full Markdown, sometimes with embedded images. Project only the columns you need for memory-constrained workloads.398- **Timeline data varies by event type.** The `data` field in `timeline_events` contains the raw event payload as JSON. The schema depends on `event_type`.399 400## Personal and sensitive information401 402Usernames, user IDs, and author associations are included as they appear in the GitHub API. All data was already publicly accessible on GitHub. Email addresses do not appear in this dataset (they exist only in git commit objects, which are in the separate code archive, not here). No private repository data is present.403 404## License405 406Released under the [Open Data Commons Attribution License (ODC-By) v1.0](https://opendatacommons.org/licenses/by/1-0/). The underlying data is sourced from GitHub's public API. [GitHub's Terms of Service](https://docs.github.com/en/site-policy/github-terms/github-terms-of-service) apply to the original data.407 408## Thanks409 410All the data here comes from [GitHub](https://github.com/)'s public [REST API](https://docs.github.com/en/rest) and [GraphQL API](https://docs.github.com/en/graphql). We are grateful to the open-source maintainers and contributors whose work is represented in these tables.411 412- **[OpenGitHub](https://huggingface.co/datasets/open-index/open-github)**, our companion dataset covering the full GitHub event stream via [GH Archive](https://www.gharchive.org/) by [Ilya Grigorik](https://www.igvita.com/)413- Built with [DuckDB](https://duckdb.org/) (Go driver), [Apache Parquet](https://parquet.apache.org/) (Zstd compression), published via [Hugging Face Hub](https://huggingface.co/)414 415Questions, feedback, or issues? Open a discussion on the [Community tab](https://huggingface.co/datasets/open-index/open-github-issues/discussions).416