haidang2405/seqevalbench-exact-propensity-logged-traces
SeqEvalBench: Exact-Propensity Logged Traces SeqEvalBench is a finite clean-room benchmark for support-aware sequential off-policy evaluation (OPE). It contains 4,096 paired-seat, two-step episodes, complete proposal queues through STOP, replayable integer state transitions and accounting, and exact rational behavior/target likelihood components for five policies. The main research object is not an estimator leaderboard. It is an auditable logged-feedback system in which one… See the full description on the dataset page: https://huggingface.co/datasets/haidang2405/seqevalbench-exact-propensity-logged-traces.
SeqEvalBench: Exact-Propensity Logged Traces
SeqEvalBench is a finite clean-room benchmark for support-aware sequential off-policy evaluation (OPE). It contains 4,096 paired-seat, two-step episodes, complete proposal queues through STOP, replayable integer state transitions and accounting, and exact rational behavior/target likelihood components for five policies.
The main research object is not an estimator leaderboard. It is an auditable logged-feedback system in which one broad logger identifies a frozen development OPE task, while a restricted logger deliberately exposes positivity failures. Unsupported target/logger cells must produce named refusals instead of estimates created by implicit clipping or smoothing.
This is wholly authored synthetic data. It is not market data, a calibrated deployment simulator, evidence that one policy is superior, or a substitute for evaluation on real systems.
Load a scientific table
The six Parquet tables have heterogeneous schemas and are therefore published as separate Hugging Face subsets. transitions is the default.
from datasets import load_dataset
transitions = load_dataset(
"haidang2405/seqevalbench-exact-propensity-logged-traces",
"transitions",
split="data",
)
likelihoods = load_dataset(
"haidang2405/seqevalbench-exact-propensity-logged-traces",
"target_likelihoods",
split="data",
)Read `release/DATA_CARD.md`, `release/schema_registry.json`, and `release/feature_ledger.csv` before modeling. Only eleven frozen pre-decision focal-state fields are authorized as default policy features; every other field is explicitly denied by the ledger.
Frozen development task
The included OPE walkthrough evaluates target_buy_tilt using traces from behavior_broad. Every probability, importance weight, estimate, and diagnostic is computed with reduced integer numerator/denominator components. The independent unit is a coordinate_id cluster containing both physical seats—not either seat-specific episode row.
The paired restricted logger has zero behavior mass on some target-supported actions. The release records the positive target mass outside behavior support and the reason TARGET_POSITIVE_BEHAVIOR_ZERO. Those cells are intentional refusal tests, not missing values to impute.
Run the two CPU-only notebooks in release/notebooks/ for the validation-first Quick Start and exact-arithmetic OPE analysis.
Integrity and reproducibility
The directory release/ is a byte-identical copy of immutable release seqevalbench-public-trace-v0.1.0-193375d1fc1de1b5. Its transfer root is:
193375d1fc1de1b5db38df4163b54f0411c34c8fbfa1cea681980594a55a1349The release contains 31 files and 11,934,951 bytes. checksums.sha256, content_manifest.json, quality_report.json, READY.json, and source_manifest.json bind byte integrity, schemas, row counts, semantic hashes, and authored source inputs. Automated verification is not represented as independent human peer review.
Rights, attribution, and limitations
Generated data tables and authored documentation are released under CC BY 4.0. Code and notebook sources are Apache-2.0. The exact boundary and suggested citation are in `release/ATTRIBUTION.md`; complete license texts are preserved in the release.
The horizon is two, the state/action mechanisms are authored, no real outcomes are represented, and the release intentionally provides no oracle value for an accuracy claim. See `release/RELATED_WORK.md` for the scientific positioning and `release/PROVENANCE.md` for build custody and exclusions.