zhangdw/skills-2m

<div align="center">

🧠 Skills-2M: GitHub-scale Agent Skills Info Atlas

🔎 A GitHub-scale metadata index of 2M+ agent skills for researchers studying the agent-skill ecosystem


</div>

<p align="center"> <b>Skills-2M</b> is a large-scale SQLite index of 2M+ agent skill records collected from GitHub, normalized to help researchers study skill discovery, repository structure, metadata patterns, retrieval, and corpus construction. </p>

<p align="center"> <a href="#-quick-start"><b>Quick Start</b></a> · <a href="#-at-a-glance"><b>At a Glance</b></a> · <a href="#-schema"><b>Schema</b></a> · <a href="#-example-queries"><b>Queries</b></a> · <a href="#-responsible-use"><b>Responsible Use</b></a> </p>

<p align="center"> <img src="assets/skills-2m-growth.png" alt="Agent skills growth trend" width="100%"> </p>

[!IMPORTANT] 🧾 This dataset is an information index, not a bundled source-code archive. It stores normalized metadata, GitHub locations, names, descriptions, and crawl-derived fields for agent skills. Individual upstream repositories, skill files, and project contents remain governed by their original licenses and terms.

[!NOTE] 🌐 The dataset is designed as a high-coverage map of the agent-skill ecosystem. It is useful for discovery and analysis, but entries may contain crawler noise, stale links, duplicated naming conventions, or upstream projects that changed after the snapshot.

✨ Why Skills-2M?

Agent skills are becoming an important research object for understanding how AI agents package reusable capabilities, instructions, scripts, references, examples, and workflow knowledge. However, these skills are scattered across many GitHub repositories and are difficult to study at scale without a normalized metadata layer.

Skills-2M provides a compact metadata layer for questions like:

• —🧭 Discovery — Where are agent skills published, and how are they named?
• —🧱 Structure — Which repositories host large skill collections, and how are paths organized?
• —🔍 Retrieval — Can we rank, cluster, deduplicate, or route skills by description and repository context?
• —📈 Ecosystem analysis — Which owners, repos, branches, and metadata patterns dominate the public skill landscape?
• —🧪 Dataset construction — Which entries should be selected for downstream crawling, filtering, license review, or benchmark building?

📦 At a glance

<table> <tr> <td align="center"><b>2,119,830</b><br/>skill rows 🧠</td> <td align="center"><b>238,707</b><br/>GitHub repos 🗂️</td> <td align="center"><b>171,974</b><br/>owners 👥</td> <td align="center"><b>2,119,830</b><br/>unique GitHub URLs 🔗</td> </tr> <tr> <td align="center"><b>SQLite</b><br/><code>skills.db</code> 🗄️</td> <td align="center"><b>2.78 GiB</b><br/>local file size 💾</td> <td align="center"><b>Normalized</b><br/>metadata index 🧩</td> <td align="center"><b>Metadata only</b><br/>not source bundles ⚠️</td> </tr> </table>

Snapshot summary

🗂️ Files

🚀 Quick start

Preview first

If you only want to inspect the schema and a small slice of records, download the CSV preview first:

hf download zhangdw/skills-2m \
  --repo-type dataset \
  --include preview/skills-preview.csv \
  --local-dir skills-2m

The preview contains 1,000 deterministic rows sampled across the SQLite rowid range. It is meant for quick inspection and dataset-page display; the full corpus remains in skills.db. The preview no longer includes local index timestamps; upstream_updated_at_* is derived from metadata_json.updatedAt.

Download the full SQLite index

hf download zhangdw/skills-2m \
  --repo-type dataset \
  --include skills.db \
  --local-dir skills-2m

Inspect with SQLite

sqlite3 skills-2m/skills.db ".tables"
sqlite3 skills-2m/skills.db "SELECT COUNT(*) FROM skills;"

Use from Python

import sqlite3

conn = sqlite3.connect("skills-2m/skills.db")
conn.row_factory = sqlite3.Row

row = conn.execute("""
    SELECT
        owner_slug,
        repo_slug,
        name,
        github_url,
        json_extract(metadata_json, '$.description') AS description,
        json_extract(metadata_json, '$.stars') AS stars
    FROM skills
    ORDER BY rowid
    LIMIT 1
""").fetchone()

print(dict(row))
conn.close()

🧬 Schema

The database currently contains one main table: skills.

Primary key:

PRIMARY KEY (owner_slug, repo_slug, id)

Indexes:

CREATE INDEX idx_skills_github_url ON skills(github_url);
CREATE INDEX idx_skills_id ON skills(id);
CREATE INDEX idx_skills_owner_repo ON skills(owner_slug, repo_slug);

<details> <summary><b>🧩 Common metadata keys</b></summary>

The metadata_json field is intentionally flexible because GitHub-scale skill metadata is messy. In this snapshot, common keys include:

Note: metadata_json.updatedAt is an upstream listing/update timestamp from the source metadata, not a local SQLite insertion or crawl timestamp.

</details>

🔎 Example queries

Count skills by repository

SELECT
  owner_slug || '/' || repo_slug AS repo,
  COUNT(*) AS skill_count
FROM skills
GROUP BY owner_slug, repo_slug
ORDER BY skill_count DESC
LIMIT 20;

Search descriptions

SELECT
  name,
  owner_slug || '/' || repo_slug AS repo,
  github_url,
  json_extract(metadata_json, '$.description') AS description
FROM skills
WHERE lower(json_extract(metadata_json, '$.description')) LIKE '%browser%'
LIMIT 25;

Find high-star repositories represented in the index

SELECT
  owner_slug || '/' || repo_slug AS repo,
  MAX(CAST(json_extract(metadata_json, '$.stars') AS INTEGER)) AS stars,
  COUNT(*) AS skills
FROM skills
GROUP BY owner_slug, repo_slug
ORDER BY stars DESC
LIMIT 20;

Export a retrieval seed table

.headers on
.mode csv
.output skill_retrieval_seed.csv

SELECT
  id,
  name,
  owner_slug,
  repo_slug,
  github_url,
  json_extract(metadata_json, '$.description') AS description
FROM skills;

🧭 Suggested use cases

🧯 Scope boundaries

This dataset is intentionally lightweight: it is an index, not a complete mirror.

✅ Included:

• —Skill names and normalized identifiers
• —GitHub URLs and repository slugs
• —Branch/path metadata when available
• —JSON metadata such as descriptions, authors, stars, forks, routes, and upstream update timestamps

❌ Not included:

• —Full upstream repository contents
• —Complete skill directory archives
• —Verified license classifications for every upstream project
• —Guarantees that every URL remains live after the crawl
• —Manual quality labels or human-reviewed safety annotations

⚖️ License and rights

The original Skills-2M compilation, schema, documentation, preview sample, and chart assets are released under Creative Commons Attribution 4.0 International (CC BY 4.0). See LICENSE for the full dataset compilation license and attribution guidance.

This license does not relicense upstream GitHub repositories, skill files, source code, project documentation, descriptions, avatars, repository statistics, or other third-party/source-derived metadata. Those materials remain governed by their original licenses, terms, and rights holders.

⚖️ Responsible use

Please treat Skills-2M as a discovery and research index.

• —🪪 Check upstream licenses before redistributing, training on, or repackaging any linked content.
• —🧹 Filter before use if building a training set, benchmark, or public derivative corpus.
• —🔁 Expect drift because GitHub repositories can be renamed, deleted, rewritten, or relicensed.
• —🛡️ Avoid blind execution of scripts or code referenced by any discovered skill.
• —🧾 Preserve attribution to original repositories when presenting derived artifacts.

🧪 Verification snapshot

The current uploaded skills.db was inspected locally with SQLite before publishing this card:

integrity_check: ok
rows:            2,119,830
owners:          171,974
repos:           238,707
unique URLs:     2,119,830
file size:       2,983,604,224 bytes
schema columns:  8 (no local created_at / updated_at columns)

📚 Citation

If you use this dataset in a paper, benchmark, or data report, please cite the Hugging Face dataset page and include the snapshot date or commit hash you used.

@misc{skills2m2026,
  title        = {Skills-2M: GitHub-scale Agent Skills Info Atlas},
  author       = {Dawei Zhang},
  year         = {2026},
  howpublished = {Hugging Face Dataset},
  note         = {Large-scale SQLite metadata index of public agent skills}
}

<div align="center">

✨ Built for people mapping the next generation of reusable agent capabilities. ✨

</div>