mhmsadegh/rectom-causal-reasoning-retriever-pilot

RecToM Causal-ToM Reasoning-Aware Retriever (Pilot)

Motivation

RecToM is a dialogue-based Theory-of-Mind movie-recommendation benchmark. A prior blind multi-judge audit (project docs127) found that off-the-shelf semantic retrieval (all-MiniLM-L6-v2) returns causally incompatible cases as its top-1 hit 74-82% of the time, despite high cosine similarity -- i.e. semantic similarity between dialogues is not the same as similarity in the causal Theory-of-Mind reasoning structure required to answer the question. This model is a pilot fine-tune testing whether structure-aware supervision closes that gap, following RAR-B (Xiao et al., 2024) and RaDeR (Das et al., 2025), adapted to RecToM's causal ToM setting.

Supervision signal

Training labels come from intervention-validated causal paths (project docs126): each causal edge was tested via a real counterfactual/control intervention on the answering model, not just claimed by an LLM. Edges whose intervention outcome was REJECTED or CONFOUNDED were dropped before deriving each item's ordered relation_type sequence.

Training-data construction

• —Source pool: 80-item leakage-clean RecToM Case Bank, filtered to 56 items with a non-empty validated relation-type sequence.
• —Same-task pairs only, labeled deterministically (no LLM call):
• —MATCH: identical relation-type sequence -- 49 pairs.
• —PARTIAL: one sequence is a prefix of the other -- 81 pairs, held out, NOT used in this training run.
• —MISMATCH: sequences diverge. Hard negatives = MISMATCH pairs with frozen-baseline cosine similarity >= 0.75 (61 pairs, all kept); remaining MISMATCH pairs subsampled to 127.
• —Input text = Dialogue History + Question + Choices ONLY. Causal paths are supervision-only.

Leakage controls

• —Training pool never includes any item from RecToM's 150-item evaluation set (docs124-127).
• —Train/validation split is group-disjoint by dialogue_id, including 2 internal collision groups found and kept intact.
• —45 items in train, 11 in validation.

Backbone and training

• —Backbone: BAAI/bge-small-en-v1.5
• —Loss: ContrastiveLoss (binary MATCH/MISMATCH + mined hard negatives)
• —Epochs: 10, batch size: 8, learning rate: 2e-05, best epoch: 1
• —Seed: 42

Retrieval metrics (validation split, group-disjoint from training)

Important limitation

This is a pilot trained on a small (n=156 training pairs) intervention-validated Case Bank. It has NOT been evaluated on downstream RecToM answer-generation accuracy yet (Stage 2, pending). Metrics above measure representation/retrieval quality only, on a held-out validation split of the same small Case Bank -- not RecToM's official 150-item evaluation set, which this model has never seen in any form.