biodatageeks/vepyr_116_GRCh38_refseq

vepyr cache — Ensembl VEP 116, GRCh38 (RefSeq)

A Parquet conversion of the Ensembl VEP 116 `homo_sapiens_refseq` cache for GRCh38, for use with vepyr, a Rust/DataFusion variant annotation engine that reproduces VEP's consequence calls. It replaces VEP's Perl-serialised, gzipped cache files with columnar Parquet that DuckDB, Polars, DataFusion or Spark can read directly.

NCBI RefSeq transcripts only. The equivalent VEP invocation uses --refseq.

⚠️ This cache is not free for commercial use as a whole. It inherits COSMIC v102 and HGMD-PUBLIC 2020.4 records from the upstream VEP cache, both of which require a commercial licence. See Licensing — including how to filter those records out.

Source versions

Every source version below was read from the upstream cache's info.txt and is also embedded in the Parquet files themselves as source_* columns (see Provenance), so you can verify any of it without trusting this card.

Upstream cache: https://ftp.ensembl.org/pub/release-116/variation/indexed_vep_cache/homo_sapiens_refseq_vep_116_GRCh38.tar.gz

Contents

Total 2,351 files, ≈30.4 GiB, one Parquet file per contig per feature type (contigs include the primary assembly, scaffolds, patches and LRG regions).

The `variation/` directory holds the same data in all three repos (ensembl, refseq, merged) — same 464 shards, same file sizes, same rows, verified byte-equal at the column level. The three caches differ only in their transcript/exon/translation content. The files are nevertheless not checksum-identical: each embeds its own bio.vep.cache_source_type value in the Parquet metadata, so variation/ copied from another flavour will hash differently even though the variant data matches.

Schema

variation/ — known variants, their frequencies and clinical annotation:

chrom, start, end, allele_string, failed, variation_name, somatic,
clin_sig, clin_sig_allele, clin_sig_ref_allele, clinical_impact,
phenotype_or_disease, pubmed, minor_allele, minor_allele_freq,
clinvar_ids, cosmic_ids, dbsnp_ids, tier,
af_global_alleles/af_global_freqs,           -- 1000 Genomes phase 3
af_gnomade_alleles/af_gnomade_freqs,         -- gnomAD exomes v4.1
af_gnomadg_alleles/af_gnomadg_freqs          -- gnomAD genomes v4.1

transcript/ carries the full transcript model — sequences (cdna_seq, peptide_seq, spliced_seq, UTRs), structure (exons, cds_start/cds_end, translation_*), and identifiers/flags (gene_symbol, hgnc_id, mane_select, mane_plus_clinical, tsl, appris, ccds, swissprot, is_canonical, is_gencode_basic). translation_sift/ holds the precomputed SIFT and PolyPhen-2 matrices keyed by translation; exon/, motif/ and regulatory/ hold exon bounds, TF binding motifs and Regulatory Build features.

Coordinates are 1-based — recorded in the Parquet key/value metadata as bio.coordinate_system_zero_based = false, alongside bio.vep.cache_version = 116, bio.vep.cache_source_type = refseq and bio.vep.chromosomes.

variation/ rows carry a tier column (0 = warm, 1 = cold) and are sorted (tier, start), so probes restricted to common variants touch only the leading row groups.

Provenance is embedded in the data

exon/, transcript/, motif/ and regulatory/ each carry the upstream source versions as columns on every row — source_assembly, source_gencode, source_genebuild, source_dbsnp, source_clinvar, source_cosmic, source_hgmd_public, source_gnomade, source_gnomadg, source_src_1000genomes, source_sift, source_polyphen, source_regbuild, source_refseq. Check the table above against the data itself:

SELECT DISTINCT source_assembly, source_gencode, source_dbsnp,
                source_clinvar, source_cosmic, source_hgmd_public
FROM 'transcript/chr21.parquet';

source_cache is null throughout: every transcript is RefSeq-derived.

Usage

# whole cache
hf download biodatageeks/vepyr_116_GRCh38_refseq \
  --repo-type dataset --local-dir ./cache/116_GRCh38_refseq

# or just what you need — e.g. every entity for chr21. Each entity directory's
# chrom_manifest.json is required, so fetch it too. Repeat --include once per
# pattern: a second bare pattern is taken as a positional filename and the
# --include is silently dropped.
hf download biodatageeks/vepyr_116_GRCh38_refseq --repo-type dataset \
  --include '*/chr21.parquet' \
  --include '*/chrom_manifest.json' \
  --local-dir ./cache/116_GRCh38_refseq

A partial download keeps the whole-cache manifests (463 contigs each), which is fine: vepyr reads its cache metadata from a shard that is present and annotates the contigs you fetched. The one exception is vepyr 0.4.0, which opened the manifest's first shard (chr1) unconditionally; on that release trim each manifest to the shards on disk:

python - <<'EOF'
import json, pathlib
for m in pathlib.Path("./cache/116_GRCh38_refseq").glob("*/chrom_manifest.json"):
    entries = json.loads(m.read_text())
    m.write_text(json.dumps([e for e in entries if (m.parent / e["dataset"]).is_file()], indent=2))
EOF

Readable without vepyr:

SELECT gene_symbol, stable_id, biotype, mane_select, tsl
FROM 'transcript/chr21.parquet'
WHERE is_canonical AND biotype = 'protein_coding'
LIMIT 20;

Licensing

Ensembl states that data generated by the project is "available without restriction" and that Ensembl code is under Apache 2.0 — but also warns that "some of the data and software included in the distribution may be subject to third-party constraints. Users of the data and software are solely responsible for establishing the nature of and complying with any such restrictions." (Ensembl disclaimer)

Two of those third-party constraints are real and present in this cache:

You are responsible for your own compliance. If you are a commercial user, either obtain the relevant licences or exclude those records.

Filtering to a commercially-unencumbered subset

Both restricted sources are identifiable in variation/, so they can be dropped:

• —COSMIC records carry a non-null cosmic_ids (e.g. COSV53365562).
• —HGMD-PUBLIC records are coordinate-only — Ensembl receives positions without alleles, so they appear with the literal allele_string = 'HGMD_MUTATION'.

-- variation rows free of COSMIC- and HGMD-derived content
SELECT *
FROM 'variation/chr21.parquet'
WHERE cosmic_ids IS NULL
  AND allele_string <> 'HGMD_MUTATION';

For scale, on chr21 (14,574,753 rows) this removes 138,799 COSMIC rows and 1,602 HGMD-PUBLIC rows — about 0.96%. transcript/, exon/, translation_*/, motif/ and regulatory/ contain no COSMIC or HGMD content; they reference those versions only in the source_* provenance columns.

Intended use

This is a research and software-engineering resource, not a clinical product. Annotation output is not a diagnosis, and consequence calls depend on the transcript set you choose.

Citation

McLaren, W., Gil, L., Hunt, S. E., et al. The Ensembl Variant Effect Predictor. Genome Biology 17, 122 (2016). doi:10.1186/s13059-016-0974-4 Ensembl 2026. Nucleic Acids Research — see Ensembl citation guidance.

Please also cite the underlying sources you rely on (GENCODE, dbSNP, ClinVar, gnomAD, COSMIC, HGMD, SIFT, PolyPhen-2).

Cache generation

Regenerated 2026-08-29 for `datafusion-bio-function-vep` v0.19.2.

This revision fixes transcript de-duplication across VEP cache regions. A transcript spanning a 1 Mb cache-region boundary is stored in more than one region file, and the copies can differ in protein_features order. Ensembl VEP keeps the copy from the first region it loads (AnnotationType/Transcript.pm::merge_features, first-wins by dbID over a region-ordered feature list), so a whole-chromosome run takes the lowest region. The cache builder previously preferred the region containing the transcript start, which selects a different copy and changes the DOMAINS CSQ field.

Only translation_core changed; translation_sift is byte-identical to the previous revision.

Compatibility. The fix is in the data, not the engine, so this cache works with any engine version. Pair it with v0.19.2 or later to also get the four consequence-engine fixes released alongside it (transcript_ablation, EXON/INTRON ranges, intron_variant at short introns, and HGVSp across a dropped insertion flank).

Verification. vepyr reproduces Ensembl VEP 116 byte-identically with this cache on chr1-22, chrX and chrY — 4,096,123 autosome records plus 157,690 chrX and 63,736 chrY, strict body MD5, zero field-value and zero field-order mismatches.

Revision 2026-09-03 — reproducible builds, smaller shards

Rebuilt with the cache-build determinism fixes (bio-formats#246, bio-functions#224, #225):

• —Builds are now byte-reproducible. Two conversions of the same raw cache previously disagreed: translation_core rows were emitted in thread-arrival order, and an exon of a transcript straddling a 1 Mb cache-region boundary could be taken from either region file. Both are now fully ordered.
• —`raw_object_json`, `object_hash` and `_rn` were dropped from transcript/, exon/, regulatory/ and motif/. Nothing reads them — they were the raw Storable payload, its digest, and a deduplication helper — and they were 40–90% of those four directories. Column counts drop by 2, or 3 for exon.
• —variation/ and translation_sift/ are unchanged and byte-identical to the previous revision; they use different write paths that neither fix touches.

Annotation output is unaffected: HG002 WGS re-annotated against these caches is still byte-identical to Ensembl VEP 116 — 140/140 MD5 body digests across the three cache types.