hmellor/new-model-additions

Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference

New Model Additions

A static Space comparing the pace of new-model support in huggingface/transformers (label "New model") vs vllm-project/vllm (label new-model).

How data flows

Instead of crawling the GitHub API on every visit, the bulk/historical PR data is served from a prebuilt parquet dataset on the Hub — `hmellor/new-model-additions`. The browser:

1. 1.Loads prs.parquet + meta.json from the dataset (js/parquet.js, via hyparquet).
2. 2.Tops up from the GitHub API only for PRs merged on/after meta.cutoff (js/hooks.js), applying the vLLM file check to just that small delta.
3. 3.Merges + dedupes by PR number and caches the result in localStorage (1h).

If the dataset can't be loaded, the browser falls back to a full GitHub crawl.

Rebuilding the dataset

The dataset is regenerated manually (no CI):

pip install -r scripts/requirements.txt
GITHUB_TOKEN=ghp_xxx HF_TOKEN=hf_xxx python scripts/build_dataset.py
# or, to inspect output without uploading:
GITHUB_TOKEN=ghp_xxx python scripts/build_dataset.py --dry-run

scripts/build_dataset.py replicates the vLLM filtering from js/utils.js and the shared blacklist in data/blacklist.json, writes prs.parquet + meta.json, and uploads them to the dataset repo. If it rejects vLLM PRs that aren't yet in data/blacklist.json, it prints them so you can add them. Because the browser tops up anything merged since the last build, the dataset only needs occasional rebuilds.

GITHUB_TOKEN is used only for the vLLM per-PR file checks (the high-volume part) — searches are always sent unauthenticated. This is deliberate: the GitHub Search API ignores a fine-grained token's repository scoping and returns 422 … do not have permission to view for public repos the token isn't explicitly granted. The token still needs to be able to read public repos for the file checks; a classic PAT (no scopes needed for public read) or a fine-grained token with public-repo read both work. The same split applies in the browser: any token pasted into the UI is used only for the file checks, not for search.