saminbbkhan/vertical-event-lm-freshretail

Vertical Event LM — FreshRetailNet

A lean, field-aware decoder-only Vertical Event LM for censored-demand forecasting on FreshRetailNet-50K (arXiv:2505.16319).

Architecture thesis: tokenized daily events → decoder-only transformer → LoRA-ready adaptation → regression / quantile / decision heads, surrounded by leakage-safe measurement (calibration, bootstrap CIs, newsvendor decision cost) that the source dataset paper does not provide.

Repo layout

vel/               # library: schema, tokenizer, model, data, trainer, eval_harness, baselines
train_job.py       # training entrypoint: uses local vel/ when present, fetches vel/ in clean HF Jobs
scripts/local_preflight.py       # local no-GPU correctness gate
scripts/build_tokenized_dataset.py  # offline tokenization for fast GPU startup
DESIGN.md          # research-justified design decisions (2024-2026 SOTA)
results.json       # current checked-in checkpoint audit on the first 256 eval series
eval_predictions.parquet  # predictions matching results.json for artifact auditing
model.safetensors, config.json, tokenizer_manifest.json  # the trained artifact
outputs/           # ignored local diagnostics, tokenized data, and benchmark artifacts

Protocol (matches official Dingdong-Inc/frn-50k-baseline)

• —Hard calendar split (leakage-safe by construction): train = 90 days (2024-03-28→06-25), eval = 7 days (2024-06-26→07-02), 50,000 store-product series.
• —Target: daily sale_amount (= Σ of 24 hourly demand values). Horizon: 7 days.
• —Metrics: WAPE = Σ|ŷ−y| / Σ|y|, WPE = Σ(ŷ−y) / Σ|y|, computed per target date then averaged, scored only on rows with `stock_hour6_22_cnt == 0` (uncensored target days, so the ground truth itself is not stockout-biased). Stratified by mean training sale (psd): Overall (≥0), High (≥1), Low (<1).
• —Published anchors (paper Table 3): TFT + TimesNet demand-recovery → 29.02% WAPE; raw-sales TFT → 31.75% WAPE with −7.37% WPE (systematic underestimation from censoring).

Reference baselines (verified, this harness, 8000-series held-out eval)

Negative WPE on the naive baselines reproduces the paper's documented censoring underestimation bias — the phenomenon the censored-demand head is designed to correct.

Current status (2026-06-07)

The checked-in small checkpoint is mechanically valid after the CENSOR-position fix, but it is not a competitive trained model. results.json and eval_predictions.parquet are an artifact audit on the first 256 eval series and should be read as a consistency check, not a SOTA claim. The model remains below the simple moving-average and seasonal baselines:

The best observed local result is documented in BEST_OBSERVED.md. It is not the checked-in model, and the original /private/tmp artifact bundle is gone, but the preflight report preserved the key metrics. Keep this as the reproduction target.

No full GPU run should be treated as meaningful until the local gate passes:

python scripts/local_preflight.py --mode all

That gate includes a deterministic overfit-forecast phase: a pilot model must train locally on four real series and then forecast the same held-out 7-day window through the real autoregressive decode path with q50 WAPE <= 0.05 and first-day WAPE <= 0.02. This phase intentionally uses a simple raw-q50 MSE point-fit objective with LM, pinball, and Tobit losses disabled; it is a correctness gate for alignment and forecast mechanics, not a claim that the distributional training objective is solved.

Local evidence and quality gates

Use the zero-shot sweep to compare the Event LM against off-the-shelf forecasting foundation models under the same FreshRetailNet scoring harness. When outputs/foundation_sweep/benchmark_2000_zero_shot/summary.json is present and its ordered series-key hash matches the tokenized dataset under diagnosis, scripts/analyze_run.py and scripts/local_preflight.py treat the best foundation WAPE as a quality gate. The sweep still does not update results.json, eval_predictions.parquet, or model.safetensors; it is an external benchmark artifact that a regenerated Event LM bundle must beat before Hub push or GPU readiness is allowed. The analyzer also carries the FreshRetailNet paper Table 3 TFT+TimesNet demand-recovery reference (0.2902 WAPE) as a hard pre-GPU target: beating moving average or Chronos-2 on the 2,000-series slice is not enough if the run is still above that paper SOTA reference.

Current stable 2,000-series zero-shot benchmark:

On the same 2,000-series slice, moving average WAPE is 0.353911. The best observed local screen is the direct-horizon ensemble at WAPE 0.294794 with 100% key overlap, zero static-token OOVs, and zero artifact-metric delta. It beats Chronos-2 but still fails the FreshRetailNet paper SOTA reference (0.2902), and the original artifact directory was lost from /private/tmp. The strongest persisted model-only diagnostic in this checkout is base_point_wape_q70_w0p5_2k at WAPE 0.320620. The corrected_2k diagnostic is weaker at WAPE 0.347791.

There is also an ignored local teacher-blend screen at WAPE 0.302429. Treat it only as evidence that the direct model, Chronos-2, and SSA prior have complementary signal; it is not a trained vertical model and not a quality-gate pass. Any /tmp or /private/tmp artifact referenced by old preflight reports must be rerun or copied into a durable artifact directory before it is cited as a reproducible model bundle.

The local no-GPU unlock target is WAPE <= 0.2902 with all artifact/key/static-vocab checks passing. Beating moving average or Chronos-2 alone is no longer a sufficient claim for this project.

Next local hypothesis run

Current diagnostics show coherent data mechanics but no durable model-quality pass. The repeated weak spot is product-level error, especially high-stockout product 834, where the direct model overpredicts while Chronos-2 and official-SSA-style priors stay closer.

The next local-only experiment is therefore not another seed sweep. It tests one research-grounded hypothesis: train the direct model as a covariate-aware residual adapter around a train-only SSA / moving-average anchor, while adding per-series WAPE pressure and finite pseudo-demand pressure on censored rows. This aligns with the FreshRetailNet paper's latent-demand-recovery lesson and the recent TSFM adapter pattern: a generic/foundation prior is useful, but the vertical model has to learn bounded, covariate-aware corrections under the FreshRetail stockout protocol.

This is still a MacBook preflight run; keep Hub push disabled and do not rent GPU unless the analyzer gate passes.

Manual runs should set VEL_RUN_NAME and VEL_RUN_HYPOTHESIS; these are copied into results.json, analyzer diagnostics, and the comparison table so each run remains tied to one falsifiable idea.

Current M4 Max residual-anchor candidate. The mkdir -p matters: without it, tee can fail to open the log path before Python creates the artifact directory.

ART=/tmp/vel_ckpt_direct_horizon_residual_anchor_ssa_series_wape_seed2025
mkdir -p "$ART"

set -o pipefail
PYTHONUNBUFFERED=1 PYTORCH_ENABLE_MPS_FALLBACK=1 \
VEL_RUN_NAME=direct_horizon_residual_anchor_ssa_series_wape_seed2025 \
VEL_RUN_HYPOTHESIS="Covariate-aware residual adapter around train-only SSA/moving-average anchors, with per-series WAPE and finite pseudo-demand pressure for censored stockout rows to reduce seed variance and product-level overprediction." \
.venv/bin/python scripts/train_direct_horizon.py \
  --artifact-dir "$ART" \
  --tokenized-dir outputs/tokenized/canary-2000-static \
  --device mps \
  --seed 2025 \
  --encoder-length 70 \
  --prediction-length 7 \
  --d-model 256 \
  --num-layers 4 \
  --nhead 8 \
  --dim-feedforward 768 \
  --dropout 0.1 \
  --batch-size 256 \
  --max-steps 2500 \
  --lr 7e-4 \
  --weight-decay 1e-4 \
  --point-quantile 0.5 \
  --point-wape-loss-w 0.25 \
  --horizon-wape-loss-w 0.3 \
  --series-wape-loss-w 0.15 \
  --censored-lower-loss-w 0.15 \
  --pseudo-censored-loss-w 0.10 \
  --pseudo-censored-source max_ssa_moving_avg \
  --residual-anchor ssa_moving_avg_blend \
  --residual-anchor-ssa-w 0.7 \
  --residual-scale 0.6 \
  --validation-tail-days 0 \
  --no-select-point-on-validation \
  2>&1 | tee "$ART/train.log"

Compare completed local hypotheses with:

.venv/bin/python scripts/capture_run_diagnostics.py --name <run_name>
.venv/bin/python scripts/compare_diagnostics.py outputs/diagnostics/*diagnostics.json

Chronos can be tried locally from an isolated uv environment:

uv run --python 3.11 \
  --with pandas --with pyarrow --with numpy --with torch --with huggingface_hub \
  --with chronos-forecasting \
  scripts/run_foundation_sweep.py \
  --models chronos2_small,chronos_bolt_small \
  --max-series 2000 \
  --device auto \
  --continue-on-error

Run Moirai separately with uni2ts==2.0.0; unconstrained resolver combinations can select older Uni2TS releases that do not include Moirai 2:

uv run --python 3.11 \
  --with pandas --with pyarrow --with numpy --with huggingface_hub \
  --with 'uni2ts==2.0.0' --with gluonts \
  scripts/run_foundation_sweep.py \
  --models moirai2_small \
  --max-series 2000 \
  --device cpu \
  --continue-on-error

Toto 2.0 currently requires Python 3.12+ and two Datadog Git subpackages:

uv run --python 3.12 \
  --with pandas --with pyarrow --with numpy --with torch --with huggingface_hub \
  --with 'dd-unit-scaling @ git+https://github.com/DataDog/toto.git#subdirectory=dd_unit_scaling' \
  --with 'toto-2 @ git+https://github.com/DataDog/toto.git#subdirectory=toto2' \
  scripts/run_foundation_sweep.py \
  --models toto2_22m \
  --max-series 2000 \
  --device cpu \
  --continue-on-error

Sweep outputs are written to outputs/foundation_sweep/<run_id>/summary.md, summary.json, and per-model prediction parquet files. Moirai is included only as a research comparison because its Hugging Face weights are non-commercial licensed.

Model

Llama-architecture causal transformer (RoPE, GQA, SwiGLU). Sizes: pilot≈5M, small≈25M, base≈83M params. Three heads, all trained jointly:

1. 1.LM head — next-token cross-entropy over the event vocab (self-supervised scaffold that conditions the sequence representation).
2. 2.Quantile head — 9 equidistant quantiles {0.1…0.9}, trained as independent logits with pinball loss. This is a simplification after the failure, not the root cause of WAPE ≈ 168. Quantiles are sorted only at inference to enforce non-crossing without coupling gradients during training. Point forecasts use the raw trained q50 head; sorted quantiles are interval-reporting outputs.
3. 3.Censored-demand head — Tobit Gaussian NLL on log1p-demand. Stockout days (stock_hour6_22_cnt > 0) are treated as right-censored lower bounds on latent demand (survival term −logΦ(−z)); in-stock days use the exact Gaussian NLL.

Forecasts are produced autoregressively over the 7-day horizon using known-future covariate tokens, and read the continuous quantile/demand heads (no value-quantization error). Sensitive losses run in fp32 with autocast disabled (bf16 destabilises the censored NLL / log_ndtr). LoRA-ready via modules_to_save on the heads.

Field-aware event tokenizer

Each store-product series → a sequence of daily events. Each day serializes to 11 tokens: DOW, HOL, ACT, DISC, PREC, TEMP, HUM, WIND, CENSOR, DEMAND, EV_END, with covariate values quantized into train-fit bins (manifest is versioned + reversible). The regression heads are supervised at the CENSOR position — the last token before the demand token — so the hidden state the heads read at training time matches exactly what is available at forecast time (see Failure analysis).

Failure analysis (the kind of thing this project is meant to demonstrate)

The first full training run converged beautifully (loss → 0) yet produced WAPE ≈ 168 on held-out data — a ~450× over-prediction. Diagnosis from the stale prediction artifact + the metric pattern (WAPE ≈ WPE, coverage ≈ 0.99): a teacher-forcing leakage bug.

The regression heads were originally supervised at the DEMAND token position. In a causal LM the hidden state at position t is computed from tokens 0…t inclusive, so at the DEMAND position the model already had the true demand token in its input — the heads learned to read the answer off the input rather than predict it. At forecast time that token does not exist, so the heads produced garbage.

Fix: supervise the heads at the CENSOR position (one token earlier), which carries all of that day's covariates but not its demand — identical to the autoregressive decode. One-line change (reg_pos_in_event = 8), no change to the model, losses, or protocol. This is exactly the leakage-vs-decode-mismatch failure the eval harness was built to catch.

Non-goals until the local gates pass: scheduled sampling, demand-token noise augmentation, custom CUDA kernels, larger GPU sweeps, and Hub publication. Those are follow-ups only after a regenerated artifact beats the benchmark-aware quality gate.

Reproduce

pip install torch==2.12.0 transformers==5.9.0 datasets==4.8.5 pandas==3.0.3 pyarrow==24.0.0 huggingface_hub==1.17.0 accelerate==1.13.0 peft==0.19.1 trackio==0.26.0 hf-transfer==0.1.9
python scripts/local_preflight.py --mode static
python scripts/local_preflight.py --mode protocol
python scripts/local_preflight.py --mode overfit-forecast --overfit-series 4 --overfit-steps 800 --device auto
python scripts/build_tokenized_dataset.py --output-dir outputs/tokenized/frn50k-event-v1 --max-series 0

# config via env vars; local checkouts use ./vel, clean HF Jobs fetch vel/ from this repo
VEL_TOKENIZED_DATASET_DIR=outputs/tokenized/frn50k-event-v1 \
VEL_SIZE=small VEL_MAX_STEPS=4000 VEL_EVAL_SERIES=3000 \
python -c "import urllib.request,runpy,os; \
urllib.request.urlretrieve('https://huggingface.co/saminbbkhan/vertical-event-lm-freshretail/resolve/main/train_job.py','train_job.py'); \
runpy.run_path('train_job.py', run_name='__main__')"

See DESIGN.md for the full research justification (Chronos / TimesFM / Moirai-2 / Chronos-2 / FreshRetailNet lineage) and the v2 roadmap (weekly patching, TFT-style variable-selection gating, sinh⁻¹ normalization, two-stage TimesNet-recovery ablation).

<!-- ml-intern-provenance -->

Generated by ML Intern

This model repository was generated by ML Intern, an agent for machine learning research and development on the Hugging Face Hub.

• —Try ML Intern: https://smolagents-ml-intern.hf.space
• —Source code: https://github.com/huggingface/ml-intern