Datasetpublic

datamatastudios/skill-scarcity-index

Datamata Skill Scarcity Index Which tech skills are genuinely hard to hire for: a daily composite scarcity score per skill built from how long roles stay open (time-to-fill), the salary premium employers pay over the category median and how often the same role is re-posted after failing to fill. Computed from active job listings across public company career pages and job boards. Latest snapshot: 2026-09-22 Rows in this release: 16331 Updated: daily Licence: CC BY 4.0 — free to… See the full description on the dataset page: https://huggingface.co/datasets/datamatastudios/skill-scarcity-index.

sourceHugging Facecc-by-4.0updated 1h agoView on Hugging Face

0likes305downloads

Dataset Card

[image]

Datamata Skill Scarcity Index

Which tech skills are genuinely hard to hire for: a daily composite scarcity score per skill built from how long roles stay open (time-to-fill), the salary premium employers pay over the category median and how often the same role is re-posted after failing to fill. Computed from active job listings across public company career pages and job boards.

—Latest snapshot: 2026-09-22
—Rows in this release: 16331
—Updated: daily
—Licence: CC BY 4.0 — free to use and adapt, including commercially, with attribution.
—Source & methodology: <https://www.datamatastudios.com/datasets/skill-scarcity-index>

Quickstart

python

import pandas as pd

# Stream straight from the Hub — no download step needed
df = pd.read_csv("hf://datasets/datamatastudios/skill-scarcity-index/skill-scarcity-index.csv")

# Headline: hardest-to-hire data skills right now
latest = df[df.snapshot_date == df.snapshot_date.max()]
print(
    latest[latest.category == "data"]
    .sort_values("scarcity_score", ascending=False)
    [["skill_name", "scarcity_score", "median_days_open", "salary_premium_pct"]]
    .head(10)
)

Or load it with the 🤗 datasets library:

python

from datasets import load_dataset

ds = load_dataset("datamatastudios/skill-scarcity-index")

What you can answer with it

—Which skills keep roles open longest — and whether that gap is widening.
—Which skills command a salary premium over their category median.
—Which skills employers repeatedly fail to hire for (repost rate).
—How scarcity is shifting as AI skills flood into job requirements.

What's NOT in this file (live on the site)

The composite score is here in full. Three deeper cuts are computed from the same pipeline but only published interactively:

—Company adoption feed — the first date each company started hiring for each skill ("47 companies added Iceberg to job requirements this quarter"): <https://www.datamatastudios.com/datasets/skill-scarcity-index>
—Stack combinations — which skill triples (e.g. AWS + dbt + Snowflake) actually appear together in postings, monthly.
—Time-to-fill by role and seniority — the same lifespan metric cut by role rather than skill.

Columns

Column	Type	Description
`snapshot_date`	string	UTC date the snapshot was taken (YYYY-MM-DD).
`category`	string	Job category: data, engineering, product, devops, security or ai.
`skill_name`	string	Canonical skill name from the extraction taxonomy.
`demand_count`	number	Active listings mentioning the skill on the snapshot date.
`demand_pct`	number	demand_count as a percentage of all active listings in the category.
`median_days_open`	number	Median days recently-closed listings with this skill stayed open. Blank below the sample floor.
`salary_premium_pct`	number	Median disclosed salary of listings with this skill vs the category median, in percent. Blank below the sample floor.
`repost_rate_pct`	number	Share of this skill's listings that are re-posts of an earlier identical role (a failed-hire signal).
`scarcity_score`	number	0-100 weighted percentile-rank composite of the three components within the category. Higher = harder to hire.

How it is built

Each day we snapshot every active job listing scraped from public company career pages and job boards, extract skills with a curated taxonomy and combine three hard-to-hire signals per skill: median lifespan of closed listings (time-to-fill), median disclosed salary vs the category median and the share of listings that are re-posts of an earlier identical role. The score is a weighted percentile-rank composite within each category, so scores are comparable across categories. Full method and known limitations: <https://www.datamatastudios.com/methodology>.

Citation

Datamata Studios. "Datamata Skill Scarcity Index." 2026-09-22. https://www.datamatastudios.com/datasets/skill-scarcity-index. Licensed under CC BY 4.0.