open-index/open-github

OpenGitHub

What is it?

This dataset contains every public event on GitHub: every push, pull request, issue, star, fork, code review, release, and discussion across all public repositories. GitHub is the world's largest software development platform, home to over 200 million repositories and the daily work of tens of millions of developers, from individual open-source contributors to the engineering teams behind the most widely used software on earth.

The archive currently spans from 2015-04-14 to 2015-08-09 (118 days), totaling 60,724,151 events across 16 fully structured Parquet tables. New events are fetched directly from the GitHub Events API every few seconds and committed as 5-minute Parquet blocks through an automated live pipeline, so the dataset stays current with GitHub itself.

We believe this is the most complete and regularly updated structured mirror of public GitHub activity available on Hugging Face. The original 19.0 GB of raw GH Archive NDJSON has been parsed, flattened, and compressed into 9.8 GB of Zstd-compressed Parquet. Every nested JSON field is expanded into typed columns — no JSON parsing needed downstream. The data is partitioned as data/TABLE/YYYY/MM/DD.parquet, making it straightforward to query with DuckDB, load with the datasets library, or process with any tool that reads Parquet.

The underlying data comes from GH Archive, created by Ilya Grigorik, which has been recording every public GitHub event via the Events API since 2011. Released under the Open Data Commons Attribution License (ODC-By) v1.0.

Live data (today)

Events from today are captured in near-real-time from the GitHub Events API and stored as 5-minute blocks in today/raw/YYYY/MM/DD/HHMM.parquet. Each block contains a generic event record with the full JSON payload preserved for later processing. Live blocks are committed to this dataset within minutes of the events occurring.

Live event schema

# Query today's live events with DuckDB.
# Run: uv run live_events.py
import duckdb

duckdb.sql("""
  SELECT event_type, COUNT(*) as n
  FROM read_parquet('hf://datasets/open-index/open-github/today/raw/**/*.parquet')
  GROUP BY event_type ORDER BY n DESC
""").show()

Events per year

  2015  ██████████████████████████████  60.7M

Pushes per year

Pushes are the most common event type, representing roughly half of all GitHub activity. Each push can contain multiple commits. Bots (Dependabot, Renovate, CI pipelines) account for a significant share.

  2015  ██████████████████████████████  29.4M

-- Top 20 repos by push volume this year.
-- Run: duckdb -c ".read pushes_top_repos.sql"
SELECT repo_name, COUNT(*) as pushes, SUM(size) as commits
FROM read_parquet('hf://datasets/open-index/open-github/data/pushes/2026/**/*.parquet')
GROUP BY repo_name ORDER BY pushes DESC LIMIT 20;

Issues per year

Issue events track the full lifecycle: opened, closed, reopened, labeled, assigned, and more. Use the action column to filter by lifecycle stage.

  2015  ██████████████████████████████  2.8M

-- Repos with the most issues opened vs closed this year.
-- Run: duckdb -c ".read issues_top_repos.sql"
SELECT repo_name,
  COUNT(*) FILTER (WHERE action = 'opened') as opened,
  COUNT(*) FILTER (WHERE action = 'closed') as closed
FROM read_parquet('hf://datasets/open-index/open-github/data/issues/2026/**/*.parquet')
GROUP BY repo_name ORDER BY opened DESC LIMIT 20;

Pull requests per year

Pull request events cover the full review cycle: opened, merged, closed, review requested, and synchronized (new commits pushed). The merged field indicates whether a PR was merged when closed.

  2015  ██████████████████████████████  3.0M

-- Top repos by merged PRs this year.
-- Run: duckdb -c ".read prs_top_merged.sql"
SELECT repo_name, COUNT(*) as merged_prs
FROM read_parquet('hf://datasets/open-index/open-github/data/pull_requests/2026/**/*.parquet')
WHERE action = 'merged'
GROUP BY repo_name ORDER BY merged_prs DESC LIMIT 20;

Stars per year

Stars (WatchEvent in the GitHub API) reflect community interest and discovery. Starring patterns often correlate with Hacker News, Reddit, or Twitter posts. For 2012–2014 events, repo_language, repo_stars_count, and repo_forks_count are populated from the legacy Timeline API repository snapshot.

  2015  ██████████████████████████████  5.4M

-- Most starred repos this year.
-- Run: duckdb -c ".read stars_top_repos.sql"
SELECT repo_name, COUNT(*) as stars
FROM read_parquet('hf://datasets/open-index/open-github/data/stars/2026/**/*.parquet')
GROUP BY repo_name ORDER BY stars DESC LIMIT 20;

Quick start

Python (datasets)

# Quick-start: load OpenGitHub data with the Hugging Face datasets library.
# Run: uv run quickstart_datasets.py
from datasets import load_dataset

# Stream all stars without downloading everything
ds = load_dataset("open-index/open-github", "stars", streaming=True)
for row in ds["train"]:
    print(row["repo_name"], row["actor_login"], row["created_at"])
    break  # remove to stream all

# Load a specific month of issues
ds = load_dataset("open-index/open-github", "issues",
                   data_files="data/issues/2026/03/*.parquet")
print(f"March 2026 issues: {len(ds['train'])}")

# Load all pull requests into memory
ds = load_dataset("open-index/open-github", "pull_requests")
print(f"Total PRs: {len(ds['train'])}")

# Query today's live events
ds = load_dataset("open-index/open-github", "live", streaming=True)
for row in ds["train"]:
    print(row["event_type"], row["repo_name"], row["created_at"])
    break  # remove to stream all

DuckDB

-- Quick-start DuckDB queries for the OpenGitHub dataset.
-- Run: duckdb -c ".read quickstart.sql"

-- Top 20 most-starred repos this year
SELECT repo_name, COUNT(*) as stars
FROM read_parquet('hf://datasets/open-index/open-github/data/stars/2026/**/*.parquet')
GROUP BY repo_name ORDER BY stars DESC LIMIT 20;

-- Most active PR reviewers (approvals only)
SELECT actor_login, COUNT(*) as approvals
FROM read_parquet('hf://datasets/open-index/open-github/data/pr_reviews/2026/**/*.parquet')
WHERE review_state = 'approved'
GROUP BY actor_login ORDER BY approvals DESC LIMIT 20;

-- Issue open/close rates by repo
SELECT repo_name,
  COUNT(*) FILTER (WHERE action = 'opened') as opened,
  COUNT(*) FILTER (WHERE action = 'closed') as closed,
  ROUND(COUNT(*) FILTER (WHERE action = 'closed') * 100.0 /
    NULLIF(COUNT(*) FILTER (WHERE action = 'opened'), 0), 1) as close_pct
FROM read_parquet('hf://datasets/open-index/open-github/data/issues/2026/**/*.parquet')
WHERE is_pull_request = false
GROUP BY repo_name HAVING opened >= 10
ORDER BY opened DESC LIMIT 20;

-- Full activity timeline for a repo (one month)
SELECT event_type, created_at, actor_login
FROM read_parquet('hf://datasets/open-index/open-github/data/*/2026/03/*.parquet')
WHERE repo_name = 'golang/go'
ORDER BY created_at DESC LIMIT 100;

Bulk download (huggingface_hub)

# Download OpenGitHub data locally with huggingface_hub.
# Run: uv run quickstart_download.py
# For faster downloads: HF_HUB_ENABLE_HF_TRANSFER=1 uv run quickstart_download.py
from huggingface_hub import snapshot_download

# Download only stars data
snapshot_download("open-index/open-github", repo_type="dataset",
                  local_dir="./open-github/",
                  allow_patterns="data/stars/**/*.parquet")

# Download a specific repo's data across all tables
# snapshot_download("open-index/open-github", repo_type="dataset",
#                   local_dir="./open-github/",
#                   allow_patterns="data/*/2026/03/*.parquet")

For faster downloads, install pip install huggingface_hub[hf_transfer] and set HF_HUB_ENABLE_HF_TRANSFER=1.

Schema

Event envelope (shared across all 16 tables)

Every row includes these columns:

Per-table payload fields

pushes.PushEvent

Git push events, typically the highest volume table (~50% of all events). Each push includes the full list of commits with SHA, message, and author.

Processing: Each PushEvent produces one row. The commits field is a Parquet LIST of structs with fields sha, message, author_name, author_email, distinct, url. All other fields are flattened directly from payload.*.

issues.IssuesEvent

Issue lifecycle events: opened, closed, reopened, edited, labeled, assigned, milestoned, and more. Contains the full issue snapshot at event time.

Processing: Flattened from payload.issue.*. Nested objects like issue.user become user_login, issue.milestone becomes milestone_id/milestone_title. Labels and assignees are Parquet LIST columns.

issue_comments.IssueCommentEvent

Comments on issues and pull requests. Each event contains both the comment and a summary of the parent issue.

Processing: Flattened from payload.comment.* and payload.issue.*. Comment reactions are flattened from comment.reactions.*. The parent issue fields are prefixed with issue_ for context.

pull_requests.PullRequestEvent

Pull request lifecycle: opened, closed, merged, labeled, review_requested, synchronize, and more. The richest table, containing diff stats, merge status, head/base refs, and full PR metadata.

Processing: Deeply flattened from payload.pull_request.*. Branch refs like head.ref, head.sha, base.ref become head_ref, head_sha, base_ref. Repository info from head.repo and base.repo become head_repo_full_name, base_repo_full_name. Labels and reviewers are Parquet LIST columns.

pr_reviews.PullRequestReviewEvent

Code review submissions: approved, changes_requested, commented, or dismissed. Each review is one row.

Processing: Flattened from payload.review.* and payload.pull_request.*. The review state (approved/changes_requested/commented/dismissed) is the most useful field for analyzing review patterns.

pr_review_comments.PullRequestReviewCommentEvent

Line-level comments on pull request diffs. Includes the diff hunk for context and threading via in_reply_to_id.

Processing: Flattened from payload.comment.* and payload.pull_request.*. The diff_hunk field contains the surrounding diff context. Thread replies reference the parent comment via in_reply_to_id.

stars.WatchEvent

Repository star events. Who starred which repo, and when. GitHub API quirk: the event is called WatchEvent but means starring. Action is always "started" so it is not stored.

Processing: The WatchEvent payload carries no useful fields — all signal is in the event envelope (actor, repo, timestamp). For 2012–2014 events the legacy Timeline API included a full repository snapshot, so repo_language, repo_stars_count, repo_forks_count, repo_description, and repo_is_fork are populated for that era. actor_type is also populated from the legacy actor_attributes object. For 2015+ events those fields are empty; actor_avatar_url is populated instead.

forks.ForkEvent

Repository fork events. Contains metadata about the newly created fork, including its language, license, and star count at fork time.

Processing: Flattened from payload.forkee.*. The forkee is the newly created repository. Owner info from forkee.owner becomes forkee_owner_login. License from forkee.license becomes forkee_license_key. Topics are a Parquet LIST column.

creates.CreateEvent

Branch, tag, or repository creation. The ref_type field distinguishes between them.

Processing: Direct mapping from payload.* fields. When ref_type is "repository", the ref field is null and description contains the repo description.

deletes.DeleteEvent

Branch or tag deletion. Repositories cannot be deleted via the Events API.

Processing: Direct mapping from payload.* fields.

releases.ReleaseEvent

Release publication events. Contains the full release metadata including tag, release notes, and assets.

Processing: Flattened from payload.release.*. Author info from release.author becomes release_author_login. Assets are a Parquet LIST of structs. Reactions flattened from release.reactions.*.

commit_comments.CommitCommentEvent

Comments on specific commits. Can be on a specific file and line, or on the commit as a whole.

Processing: Flattened from payload.comment.*. When the comment is on a specific file, path and line are populated. Reactions flattened from comment.reactions.*.

wiki_pages.GollumEvent

Wiki page creates and edits. A single GollumEvent can contain multiple page changes, so we emit one row per page (not per event).

Processing: The payload.pages array is unpacked: each page in the array produces a separate row, all sharing the same event envelope. This means one GitHub event can generate multiple rows.

members.MemberEvent

Collaborator additions to repositories.

Processing: Flattened from payload.member.*. The actor is who added the member; the member fields describe who was added.

public_events.PublicEvent

Repository visibility changes from private to public. The simplest table, containing only the event envelope (who, which repo, when) with no additional payload columns.

Processing: No payload fields are extracted. The event envelope alone captures the relevant information.

discussions.DiscussionEvent

GitHub Discussions lifecycle: created, answered, category_changed, labeled, and more. Includes category, answer status, and full discussion metadata.

Processing: Flattened from payload.discussion.*. Category info from discussion.category becomes category_name/category_slug/category_emoji. Answer info becomes answer_html_url/answer_chosen_at. Labels are a Parquet LIST column. Reactions flattened from discussion.reactions.*.

Per-table breakdown

How it's built

The pipeline has two modes that work together:

Archive mode processes historical GH Archive hourly dumps in a single pass per file: download the .json.gz, decompress and parse each JSON line, route by event type to one of 16 handlers, flatten nested JSON into typed columns, write to Parquet with Zstd compression, and publish daily to HuggingFace.

Live mode captures events directly from the GitHub Events API in near-real-time. Multiple API tokens poll concurrently with adaptive pagination (up to 300 events per cycle). Events are deduplicated by ID, bucketed into 5-minute blocks by their created_at timestamp, and written as Parquet files. Each block is pushed to HuggingFace immediately after writing. On each hour boundary, the corresponding GH Archive file is downloaded and merged into the typed daily tables for complete coverage.

All scalar fields are fully flattened into typed columns. Variable-length arrays (commits, labels, assets, topics, assignees) are stored as native Parquet LIST columns — no JSON strings. All *_at timestamp fields use the Parquet TIMESTAMP type (UTC microsecond precision), so DuckDB, pandas, Spark, and the HuggingFace viewer all read them as native datetimes.

No events are filtered. Every public event captured by GH Archive appears in the corresponding table. Events with parse errors are logged and skipped (typically less than 0.01%).

Known limitations

• —Full coverage starts 2015-01-01. Events from 2011-02-12 to 2014-12-31 are included but parsed from the deprecated Timeline API format, which has less detail for some event types.
• —Bot activity. A significant fraction of events (especially pushes and issues) are generated by bots such as Dependabot, Renovate, and CI systems. No bot filtering is applied.
• —Event lag. GH Archive captures events with a small delay (roughly minutes). Events during GitHub outages may be missing.
• —Pre-2015 limitations. IssuesEvent and IssueCommentEvent from 2012-2014 contain only integer IDs (no title, body, or state) because the old API did not include full objects in event payloads.

Personal information

All data was already public on GitHub. Usernames, user IDs, and repository information are included as they appear in the GitHub Events API. Email addresses may appear in commit metadata within PushEvent payloads (from public git commit objects). No private repository data is present.

License

Released under the [Open Data Commons Attribution License (ODC-By) v1.0](https://opendatacommons.org/licenses/by/1-0/). The underlying data is sourced from the public GitHub Events API via GH Archive. GitHub's Terms of Service apply to the original data.

Credits

• —[GH Archive](https://www.gharchive.org/) by Ilya Grigorik, the foundational project that has recorded every public GitHub event since 2011
• —[GitHub Events API](https://docs.github.com/en/rest/activity/events), the source data stream
• —Built with Apache Parquet (Go), published via HuggingFace Hub

Contact

Questions, feedback, or issues? Open a discussion on the Community tab.