chibifire/vast-market-snapshots

vast-market-snapshots

Canonical dataset location: chibifire/vast-market-snapshots on Hugging Face.

This GitHub repo carries the capture code only (snapshot.py + this README). Parquet payloads live on HF per the workspace's weights-live-on-huggingface rule. snapshots/ is gitignored here; run snapshot.py locally to produce a snapshot directory, then upload to the HF dataset repo with huggingface_hub.

Point-in-time snapshots of the Vast.ai on-demand GPU offer market, kept as zstd Parquet in Essential Tuple Normal Form per CLAUDE.md (interned vocabularies, satellite relations for missing values, no nullable columns, no derivable columns).

Layout

snapshots/<iso-ts>/ manifest.json # capture time, source, sha256 per file offer.parquet # core entity, one row per offer, no nulls vocab<name>.parquet # (id, value) for each interned vocabulary offer<sub>.parquet # satellite relation: (offer_id, ...) — a # row exists iff every named field was # present on the source offer

Interned vocabularies: gpu, cpu, cpu_arch, gpu_arch, disk, geo, driver, host. Satellite relations: offer_hostname, offer_geo, offer_driver, offer_cpu, offer_disk_model, offer_gpu_arch, offer_inet, offer_discount, offer_pci, offer_dlperf, offer_bw, offer_ports. Any satellite whose rows all had missing values is omitted for that snapshot rather than written empty.

Dropped derivable columns (recompute on read from the fields that remain): dph_total_adj, discounted_dph_total, discounted_hourly, dlperf_per_dphtotal, flops_per_dphtotal, cpu_cores_effective.

Capture

# needs an authenticated Vast API key at ~/.vastaiapikey python3 snapshot.py

The snapshot script is idempotent per-second — two runs in the same second would collide on the directory name, so run it at most once per second.

Read

import pyarrow.parquet as pq offer = pq.readtable("snapshots/<ts>/offer.parquet").topandas() vocabgpu = pq.readtable("snapshots/<ts>/vocabgpu.parquet").topandas() joined = offer.merge(vocabgpu.rename(columns={"id":"gpuid","value":"gpuname"}), on="gpuid", how="left")

Licence

MIT. The underlying Vast market data is public; no PII, no scraping under credentials someone else owns.