alihassan1437/causalrec-bench

CausalRec-Bench

A fully synthetic, multi-domain (e-commerce and streaming) recommendation interaction dataset generated from a documented, exactly reproducible structural model. Every click carries literal counterfactual confounder attribution — derived by reusing the same random draw that generated the observed outcome to test which confounders were causally necessary — rather than a post-hoc magnitude heuristic.

Paper: CausalRec-Bench: A Synthetic Benchmark with Counterfactual Confounder Ground Truth for Causal Recommendation Evaluation. Ali Hassan, Yan Zhang (Professor / Contributor).

Scale: 50,000 users, 4,000 items (2,000 e-commerce + 2,000 streaming), 1,341,991 logged interactions.

Why this dataset exists

Offline recommendation evaluation relies on logged clicks that conflate genuine user preference with confounding mechanics (exposure, position, promotion, popularity), and current evaluation practice largely measures whether a model reproduces past biased exposure patterns rather than whether it recovers true preference. Synthetic data with known confounder ground truth is an accepted way to test debiasing methods against this gap, but it is typically reinvented inside individual papers' appendices rather than released as a standardized, reusable public artifact. This dataset is released as a standalone, documented, versioned benchmark filling that gap — playing the role for causal-attribution-based evaluation that Yahoo!R3/Coat/KuaiRand play for randomization-based evaluation.

Structural model

Let g denote genuine_match for a given impression (a demographic-segment preference component plus a per-user idiosyncratic latent taste perturbation), let 1[·] denote an indicator function, and let ρ(p) denote the position-decay function mapping a position p ∈ {1, ..., 10} to its bias contribution (0.25 at p=1, down to 0.01 at p=9–10). Click probability for each exposed impression is:

P(click) = clip( 0.8*g

• —0.15 * 1[promoted]
• —0.12 * 1[popular]
• —ρ(p)
• —0.10 1[cold_start] 1[promoted] , 0.01, 0.95 )

A single draw r ~ Uniform(0,1) per impression determines the observed click (clicked = 1[r < P(click)]). The same draw r is reused to evaluate three counterfactual click probabilities, each with one confounder removed (e.g. P(click | do(¬promotion)) = clip(P(click) - 0.15*1[promoted], 0.01, 0.95)), producing the would_click_* columns below. A click is labeled genuine_preference if it would have occurred from preference alone; otherwise it is attributed to whichever single confounder's removal would have flipped the outcome, or labeled mixed if more than one confounder was independently sufficient.

The exact generator (generate_dataset.py, included in this repo) reproduces this dataset bit-for-bit from the documented seed, and can be reseeded to draw new instances of the same generative model.

Columns

Observable (safe to use as model input): domain, age_group, gender, income, life_stage, season, new_user, category, price_tier, popularity, promotion, item_age, position, seasonal_relevance, exposure_prob, clicked, purchased.

Ground truth / oracle (evaluation only — never use as a model feature): genuine_match, genuine_contribution, promotion_contribution, popularity_contribution, position_contribution, coldstart_contribution, click_prob, would_click_genuine_only, would_click_without_promotion, would_click_without_popularity, would_click_without_position, click_cause.

Methods evaluated on this benchmark must never have access to the ground-truth group during training or prediction.

Splits

Users are split 70/10/20 into train/validation/test; splits are disjoint over users but share the full item catalog (a cold-user generalization setting, not a temporal or random-row split). Additional cuts ship alongside the official splits: a confounding- intensity gradient (level1_simple, level2_medium, level3_hard), cold-start-only and seasonal-cold-start cuts, domain-specific cuts, and position-extremity cuts (high_position, low_position).

Validation (measured directly on this data, not asserted)

Baseline results

Five methods evaluated against the ground-truth genuine_match: Naive (no debiasing), IPS (inverse-propensity-weighted), Standardized (counterfactual standardization / regression adjustment), IPS+Standardized, and an Oracle upper bound.

Headline finding: IPS underperforms the naive baseline, because this benchmark's confounding enters additively into the outcome rather than through a missing-data selection mechanism that IPS is designed to correct. Counterfactual standardization, which is mechanistically matched to the actual confounding structure, recovers substantially more of the true preference signal (a 67.6% relative improvement over naive). Both findings are unanimous across three independent random seeds, both domains, position-extremity and seasonal cuts, and an alternative (feature-conditional) propensity construction, and are statistically significant under cluster bootstrap confidence intervals.

Repository contents

• —train.csv, val.csv, test.csv — official splits
• —level1_simple.csv, level2_medium.csv, level3_hard.csv — confounding-intensity gradient
• —cold_start.csv, {season}_cold.csv, {ecom,stream}_only.csv, {ecom,stream}_cold.csv, high_position.csv, low_position.csv — additional evaluation cuts
• —users.csv, items.csv — catalog tables
• —generate_dataset.py — the exact, reproducible generator

License

MIT.

Citation

If you use this dataset, please cite:

@misc{hassan2026causalrecbench, title = {CausalRec-Bench: A Synthetic Benchmark with Counterfactual Confounder Ground Truth for Causal Recommendation Evaluation}, author = {Hassan, Ali and Zhang, Yan}, year = {2026}, howpublished = {Hugging Face Datasets}, url = {https://huggingface.co/datasets/alihassan1437/causalrec-bench} }