LossFunctionLover/pairwise-orm-model

1---2language: en3license: apache-2.04tags:5- reward-model6- preference-learning7- agentic-reasoning8- outcome-reward-model9- pairwise-preference10datasets:11- LossFunctionLover/orm-pairwise-preference-pairs12metrics:13- accuracy14pipeline_tag: text-classification15model-index:16- name: pairwise-orm-model17  results:18  - task:19      type: preference-learning20      name: Pairwise Preference Ranking21    dataset:22      name: ORM Pairwise Preference Pairs23      type: LossFunctionLover/orm-pairwise-preference-pairs24    metrics:25    - type: accuracy26      value: 96.327      name: Pairwise Accuracy28    - type: confidence_interval29      value: "[95.3%, 97.1%]"30      name: Bootstrap 90% CI31---32 33# Pairwise Outcome Reward Model (ORM)34 35<div align="center">36 37**A Robust Preference Learning Model for Agentic Reasoning Systems**38 39[![Dataset](https://img.shields.io/badge/Dataset-HuggingFace-yellow)](https://huggingface.co/datasets/LossFunctionLover/orm-pairwise-preference-pairs)40 41</div>42 43## 📋 Model Description44 45This is a **Pairwise Outcome Reward Model (ORM)** designed for agentic reasoning systems. The model learns to rank reasoning traces through relative preference judgments rather than absolute quality scores, achieving superior stability and reproducibility compared to traditional pointwise approaches.46 47**Key Achievements:**48- ✅ **96.3% pairwise accuracy** with tight confidence intervals [95.3%, 97.1%]49- ✅ **Stable training** in just 800 optimization steps (~10 minutes on single GPU)50- ✅ **Strong anti-symmetry** (swapped accuracy: 3.75%, correlation: -0.998)51- ✅ **Calibrated uncertainty** on near-tie cases52- ✅ **Length-robust** performance (95.5% - 99.7% across token ranges)53- ✅ **Frozen base model** architecture for reproducibility54 55## 🎯 Intended Use56 57This model is designed for:58- **Best-of-N sampling** in reasoning tasks59- **Candidate ranking** in agentic search and tree-based reasoning60- **Outcome-level feedback** in multi-step reasoning systems61- **Integration with Process Reward Models (PRMs)** for comprehensive evaluation62- **Agentic frameworks** like MagiCore-Agentic for robust decision-making63 64## 🏗️ Architecture65 66```67Input Text (Reasoning Trace)68    ↓69[Frozen Base LM Encoder]  ← Pre-trained, frozen during training70    ↓71[Final Non-Padding Token Pooling (attention-mask aware)]72    ↓73[Lightweight Linear Head]    ← Only these parameters are trained74    ↓75Scalar Reward Score76```77 78**Design Philosophy:**79- **Frozen encoder**: Leverages pre-trained representations, reduces overfitting80- **Lightweight head**: <1M trainable parameters for stability81- **Minimal architecture**: Prioritizes reproducibility over complexity82 83## 📊 Training Details84 85### Dataset Construction86 87The model was trained on a carefully curated pairwise preference dataset derived from high-quality reasoning traces:88 89**Original Pointwise Dataset:**90- Train: 9,482 examples91- Validation: 524 examples  92- Test: 547 examples93- Labels: Binary (correct=1, incorrect=0)94 95**Quality Validation (Base Model Log-Probability Analysis):**96- Pearson correlation: **r = 0.87** (p < 1e-162)97- Spearman correlation: **ρ = 0.83** (p < 1e-134)98- Base model pairwise accuracy: **98.2%**99- Mean log-prob (positive): -2.17100- Mean log-prob (negative): -3.64101 102These metrics confirm strong signal separation in the base model, validating dataset quality before pairwise transformation.103 104**Pairwise Dataset Construction:**105 106The pointwise data was transformed into pairwise preferences using a global sampling strategy:107 108```python109# For each positive example, sample N negative examples110# Creates (chosen, rejected) pairs where chosen=correct, rejected=incorrect111```112 113**Dataset Statistics:**114- **Training pairs**: 41,656 (8 negatives per positive)115- **Validation pairs**: 1,144 (4 negatives per positive)116- **Test pairs**: 1,232 (4 negatives per positive)117 118Each pair contains:119- `chosen`: Correct reasoning trace (label=1)120- `rejected`: Incorrect reasoning trace (label=0)121- `meta`: Chain IDs and labels for traceability122 123**Curation Process:**124- ✅ **Weeks of manual quality control** on original dataset125- ✅ **Rigorous filtering** for correctness and reasoning quality126- ✅ **Balanced sampling** across reasoning patterns and lengths127- ✅ **Verified anti-symmetry** through base model analysis128 129### Training Configuration130 131**Hyperparameters:**132- **Base Model**: facebook/opt-1.3b133- **Trainable Parameters**: Scoring head only (~500K-1M params)134- **Optimizer**: AdamW135  - Learning rate: 2e-5136  - Betas: (0.9, 0.999)137  - Weight decay: 0.01138- **Learning Rate Schedule**: Linear warmup (50 steps) + constant139- **Batch Size**: 8 pairs140- **Gradient Clipping**: Max norm 1.0141- **Training Steps**: 800142- **Mixed Precision**: FP16143- **Hardware**: Single GPU (A100/V100)144- **Training Time**: ~10 minutes145 146**Loss Function:**147```python148# Logistic pairwise ranking loss149L = -log(sigmoid(f(x_chosen) - f(x_rejected)))150```151 152## 🔬 Evaluation Results153 154### Main Performance (Test Set: 1,232 pairs)155 156| Metric | Value |157|--------|-------|158| **Pairwise Accuracy** | **96.3%** |159| Bootstrap 90% CI | [95.3%, 97.1%] |160| Mean Margin | 1.40 |161| Median Margin | 1.52 |162| Std Deviation | 1.12 |163| Incorrect/Tied Pairs | 3.7% |164 165### Length-Based Robustness166 167| Token Range | Accuracy | Sample Size |168|-------------|----------|-------------|169| 0-128 tokens | 95.5% | 442 pairs |170| 128-256 tokens | **99.7%** | 332 pairs |171| 256+ tokens | 96.1% | 458 pairs |172 173**Key Insight**: Model does not exploit length heuristics; benefits from additional context in medium-length range.174 175### Anti-Symmetry Validation (Label-Swap Test)176 177| Metric | Value | Expected |178|--------|-------|----------|179| Swapped Accuracy | 3.75% | ~3.7% ✅ |180| Mean Swapped Margin | -1.40 | -1.40 ✅ |181| Correlation (Original vs Swapped) | -0.998 | ~-1.0 ✅ |182 183**Conclusion**: Model learns true preference ordering, not positional artifacts.184 185### Near-Tie Uncertainty Calibration186 187| Margin Threshold | Accuracy | Interpretation |188|------------------|----------|----------------|189| \|Δ\| < 0.05 | 43% | Low confidence → near chance |190| \|Δ\| < 0.10 | 48% | Uncertain region |191| \|Δ\| < 0.20 | 60% | Moderate confidence |192| \|Δ\| < 0.50 | 71% | Higher confidence |193 194**Key Insight**: Smooth calibration curve indicates well-calibrated uncertainty—critical for agentic systems that need to defer when uncertain.195 196## 💻 Usage197 198### Installation199 200```bash201pip install transformers torch huggingface_hub202```203 204### Basic Usage205 206```python207import torch208from transformers import AutoModel, AutoTokenizer209from huggingface_hub import hf_hub_download210 211# Download the trained model weights212model_path = hf_hub_download(213    repo_id="LossFunctionLover/pairwise-orm-model",214    filename="pairwise_orm.pt"215)216 217# Load the base encoder (frozen during training)218base_model = AutoModel.from_pretrained("facebook/opt-1.3b")219tokenizer = AutoTokenizer.from_pretrained("facebook/opt-1.3b")220 221# Load the trained scoring head weights222ckpt = torch.load(model_path, map_location="cpu")223state = ckpt["model_state"] if "model_state" in ckpt else ckpt224 225head_state = {226    k.replace("score.", ""): v227    for k, v in state.items()228    if k.startswith("score.")229}230 231assert set(head_state.keys()) == {"weight", "bias"}232 233# Initialize scoring head (single linear layer)234hidden_size = base_model.config.hidden_size235scoring_head = torch.nn.Linear(hidden_size, 1)236scoring_head.load_state_dict(head_state)237 238# Move to device239device = "cuda" if torch.cuda.is_available() else "cpu"240base_model.eval().to(device)241scoring_head.eval().to(device)242 243# Score a single reasoning trace244def score_trace(trace_text: str) -> float:245    """246    Compute scalar reward for a reasoning trace.247    Higher scores indicate better reasoning quality.248    """249    inputs = tokenizer(250        trace_text,251        return_tensors="pt",252        truncation=True,253        max_length=512,254        padding=True255    )256    inputs = {k: v.to(device) for k, v in inputs.items()}257    258    with torch.no_grad():259        # Get base model embeddings260        encoder_outputs = base_model(**inputs)261        # Pool at actual sequence end (accounts for padding)262        seq_lengths = inputs["attention_mask"].sum(dim=1) - 1263        pooled = encoder_outputs.last_hidden_state[torch.arange(seq_lengths.size(0)), seq_lengths]264        # Get reward score265        score = scoring_head(pooled).squeeze(-1).cpu().item()266    267    return score268 269# Example: Compare two reasoning traces270trace_1 = """2711. Calculate the cost per item: $20 / 4 = $52722. Calculate total for 10 items: $5 × 10 = $502733. Apply 10% discount: $50 × 0.9 = $45274 275Final Answer: $45276"""277 278trace_2 = """2791. Assume linear growth incorrectly2802. Multiply by unrelated constant2813. Round result arbitrarily282 283Final Answer: $38284"""285 286score_1 = score_trace(trace_1)287score_2 = score_trace(trace_2)288 289print(f"Trace 1 score: {score_1:.3f}")290print(f"Trace 2 score: {score_2:.3f}")291print(f"Preferred trace: {'Trace 1' if score_1 > score_2 else 'Trace 2'}")292print(f"Confidence (margin): {abs(score_1 - score_2):.3f}")293```294 295### Batch Scoring for Best-of-N Sampling296 297```python298def rank_candidates(candidates: list[str], return_scores: bool = False):299    """300    Rank multiple candidate reasoning traces.301    302    Args:303        candidates: List of reasoning trace strings304        return_scores: If True, return (ranked_candidates, scores)305    306    Returns:307        Ranked list of candidates (best first)308    """309    scores = [score_trace(cand) for cand in candidates]310    311    # Sort by score descending312    ranked_indices = sorted(range(len(scores)), key=lambda i: scores[i], reverse=True)313    ranked_candidates = [candidates[i] for i in ranked_indices]314    315    if return_scores:316        ranked_scores = [scores[i] for i in ranked_indices]317        return ranked_candidates, ranked_scores318    319    return ranked_candidates320 321# Example usage322candidates = [trace_1, trace_2, ...]  # Multiple traces for same problem323best_trace = rank_candidates(candidates)[0]324```325 326### Integration with Agentic Systems327 328```python329# Example: Use ORM for tree search pruning330def should_expand_node(reasoning_trace: str, threshold: float = 0.0) -> bool:331    """332    Decide whether to expand a reasoning node based on ORM score.333    """334    score = score_trace(reasoning_trace)335    return score > threshold336 337# Example: Combine with PRM for comprehensive evaluation338def hybrid_evaluation(trace: str, orm_model, prm_model):339    """340    Combine outcome-level (ORM) and process-level (PRM) rewards.341    """342    orm_score = score_trace(trace)  # Outcome quality343    prm_scores = prm_model.score_steps(trace)  # Step-level correctness344    345    # Weighted combination346    final_score = 0.5 * orm_score + 0.5 * prm_scores.mean()347    return final_score348```349 350## 🔗 Related Work & Citation351 352This work builds upon and complements:353 354- **MagiCore-Agentic** ([Liu et al., 2024](https://arxiv.org/abs/2409.12147)): Robust multi-step reasoning through agentic orchestration355- **Training Verifiers** ([Cobbe et al., 2021](https://arxiv.org/abs/2110.14168)): Math word problem verification356- **Process & Outcome Feedback** ([Uesato et al., 2022](https://arxiv.org/abs/2211.14275)): Combining reward signals357 358### Citation359 360If you use this model in your research, please cite:361 362```bibtex363@article{mishra2026pairwise-orm,364  title={Stable Outcome Reward Modeling via Pairwise Preference Learning},365  author={Mishra, Aklesh},366  year={2026},367  note={Preprint}368}369```370 371## 🔗 Resources372 373- 📄 **Paper**: Preprint374- 💾 **Dataset**: [HuggingFace](https://huggingface.co/datasets/LossFunctionLover/orm-pairwise-preference-pairs)375 376## 📧 Contact377 378**Aklesh Mishra**379- Email: akleshmishra7@gmail.com380- Independent Researcher381 382## 📝 License383 384This model is released under the **Apache 2.0 License**.385 386## 🙏 Acknowledgments387 388This research builds upon months of dedicated work in preference learning and agentic reasoning systems. Special thanks to:389 390- The **MagiCore-Agentic** team for their inspiring work on multi-step agentic reasoning391- The broader ML community for foundational research in reward modeling and RLHF392- Contributors to open-source tools (Transformers, PyTorch) that made this work possible393 394## 📊 Model Performance Summary395 396```397╔════════════════════════════════════════════════════════════╗398║  Pairwise ORM - Key Metrics                                ║399╠════════════════════════════════════════════════════════════╣400║  Pairwise Accuracy:        96.3% [95.3%, 97.1%]            ║401║  Training Steps:           800 (~10 min on single GPU)     ║402║  Dataset Quality (r):      0.87 (Pearson)                  ║403║  Anti-symmetry:            -0.998 correlation              ║404║  Length Robustness:        95.5% - 99.7% across ranges     ║405║  Uncertainty Calibration:  Smooth degradation near ties    ║406╚════════════════════════════════════════════════════════════╝407```408 409---410 411**Last Updated**: January 22, 2026