aahf/tabular-prior-probe

Tabular Prior Probe

Can a small instruct LLM make accurate row-level predictions on tabular data — or does it just recite what the column names imply?

This Space shows a completed evaluation. It runs no model and needs no GPU: all inference happened offline, and the page renders committed JSON. That is why it is a static Space — it loads instantly, never sleeps, and has no quota to exhaust.

Two runs are on the page, switchable in the header: the stock instruct model (results.json) and a LoRA fine-tune trained on aligned's training split (results_ft.json). Section 3 puts them side by side.

The instrument

Three datasets share one schema and one set of marginals, and differ only in semantics:

Prior-only AUC is what a predictor scores using nothing but what the column names imply, with no access to the data. It is the number that matters: on aligned, beating 0.5 proves nothing, because name-reasoning alone already reaches 0.81.

inverted is the sharp instrument. A model applying world knowledge lands far below chance there, because that knowledge is now precisely wrong. A model reading the few-shot rows holds above 0.5. There is no ambiguity in that gap — which is the entire reason a single-dataset benchmark cannot answer this question.

Because the data is generated and never published, there is no memorised counterpart. That is the reason not to benchmark on Titanic or Adult: a good score there measures recall.

Four conditions per dataset

• —`real` — real column names, correct few-shot labels. The headline, and alone the least informative.
• —`anon` — columns renamed col_1…, categories to cat_1…. The gap from real is the value of prior knowledge.
• —`shuffled` — few-shot examples kept, labels permuted. If this doesn't hurt, the model isn't reading them.
• —`prior` — features stripped, schema only. The floor.

Reading the fine-tune honestly

The fine-tune was trained on aligned and is therefore in-distribution on aligned. A gain there is fit, not evidence — it says nothing about whether the model learned to read rows. inverted and random are the tests that carry information, because the fine-tune never saw those semantics. If fine-tuning taught it to use the few-shot rows, it holds up there too; if it merely absorbed aligned's conclusions, inverted gets worse than the stock model, because now the wrong prior is baked into the weights rather than merely suggested by the column names.

What the two runs showed

Three things follow, and one of them is a warning about the probe itself.

It generalised. The gains are largest on inverted (+0.129), a variant the fine-tune never saw. Prior-absorption would have produced the opposite — a model that had merely internalised aligned's conclusions would do worse on inverted, because the wrong prior would then sit in the weights instead of merely being implied by the column names.

Column names stopped mattering. The real-minus-anon gap, which is what prior knowledge was worth, fell from 0.183 to 0.011.

But it is no longer doing in-context learning, and the probe's own controls will mislead you here. On inverted with the few-shot labels permuted, the fine-tune still scores 0.749 against 0.783 with correct labels — destroying the examples costs it 0.035. For a stock model a near-chance shuffled score is the proof that it is reading the rows; for a fine-tuned model a high one means the mapping has moved into the weights. run_evals.py derives its verdict from real AUC alone and never inspects shuffled, so it labels this run "genuine in-context learning". That label is wrong, and section 7 on the page says so.

Fine-tuning did not beat the classical baselines off-distribution: logistic regression still leads on inverted (0.806 vs 0.783) and random (0.821 vs 0.806).

Reproducing

python make_dataset.py --n 3000 --seed 0          # the three variants + ground truth
vllm serve openbmb/MiniCPM5-2B --port 8000        # or Ollama / LM Studio
python run_evals.py --data-dir . --model openbmb/MiniCPM5-2B \
    --n-test 300 --n-shots 16 --out results.json

# the fine-tuned run, same probe, same seed
python run_evals.py --backend local --model models/minicpm5-2b-aligned-ft \
    --n-test 300 --n-shots 16 --out results_ft.json

Commit the new JSON beside index.html and the Space picks it up on rebuild. index.html reads results.json and, if present, results_ft.json — both in the same schema, so anything producing that shape can drive the page. Drop results_ft.json and the comparison section hides itself and the page renders as a single-run report.

Caveats

• —Probabilities come from top_logprobs on the first answer token, renormalised over the label vocabulary — not from parsing generated text. Parsed labels give accuracy; AUC and calibration need a distribution.
• —Hybrid-reasoning models must have thinking disabled. Because only the first token is scored, a model that opens its turn with <think> spends that token on it and the label is never read: the yes/no mass sits ~25 nats down in the tail and the resulting AUC is a ranking of noise that looks entirely plausible (~0.57–0.69). The runner passes enable_thinking=False by default and warns when label tokens are missing from the returned distribution — do not disable that warning.
• —Bootstrap CIs on AUC differences run ±0.06 at a few hundred test rows. A two-point gap is not a result.
• —The generated truth is mostly additive, so logistic regression matches or slightly beats the GBDT at 3k rows. Real data with heavy interaction structure favours trees more.