ianlee1996/pokerbench-qwen3-14b-lora-dpo

PokerBench Qwen3-14B + LoRA + DPO (UTG-only training)

DPO checkpoint on top of `ianlee1996/pokerbench-qwen3-14b-lora-mixed`, trained on 1167 counterfactual-EV preference pairs extracted from 5000 hands of self-play (ckpt-mixed vs openai.gpt-oss-120b-1:0 on AWS Bedrock). Targets the PokerBench paper's Future Work — making the SFT model robust against non-GTO opponents.

⚠️ Known limitation: this checkpoint was trained with hero=UTG only (the first 3 of the 6 seats), and shows position-dependent generalization issues (see Stage 4 results below). A successor checkpoint with hero=all-6-seats is in progress. Use pokerbench-qwen3-14b-lora-mixed (the SFT base) instead if you need balanced position play.

Three-stage training stack

This is the third LoRA layer in a stacked SFT → SFT → DPO pipeline, all trained on top of Qwen/Qwen3-14B:

1. 1.Stage 1 SFT (04_qwen3_14b_lora_full_data_1epoch/checkpoint-4375): full PokerBench paper format, 1 epoch on 60k+500k samples → EM 90.07%
2. 2.Stage 2 SFT (06_qwen3_14b_lora_mixed_50_50/checkpoint-250): 50/50 paper + production-PE format, 250 steps → EM 89.86%, PE AA 84.0%
3. 3.Stage 3 DPO (this checkpoint): 1167 counterfactual-EV preference pairs from self-play vs gpt-oss-120b → see results below

Headline numbers (4-stage evaluation)

Position-conditional finding: the per-seat breakdown of Stage 4 reveals the DPO model wins at UTG (the only position trained, +31.70 bb/100) but loses at CO (untrained, −25.87 bb/100). The LoRA's preferences don't generalize across seats — a known consequence of training only one hero position.

Stage 4 per-seat bb/100 (DPO Team A vs Stage 2 Team B)

Method (DPO recipe)

Self-play data collection

• —5000 hands of 6-max NLHE: ckpt-mixed × 3 seats (UTG, CO, SB) vs openai.gpt-oss-120b-1:0 × 3 seats (HJ, BTN, BB)
• —Captured every decision (state, action, available moves, prompt) per seat
• —Team A (ckpt-mixed) +46.10 bb/100 vs Team B (gpt-oss-120b), reproducing the Stage 3 baseline (±1 BB)
• —Wallclock: ~9 hours / Bedrock spend: ~$50

Counterfactual EV preference extraction

For each hand, pick one key hero decision (rules: river > turn > flop > preflop ≥ 2BB), generate 2 candidate actions (low-temp + high-temp + retry, with structural fallback), and run N=10 monte-carlo rollouts per candidate with deepseek.v3.2 driving every other seat to showdown. The action with higher EV becomes chosen; the other becomes rejected. Pairs with EV gap < 0.5 BB or fewer than 5 valid MC samples are dropped.

• —1167 pairs from 5000 hands (~23% pair rate)
• —EV gap median 2.10 BB, p75 4.76 BB, max 63.95 BB
• —All pairs are hero=UTG (paper seat 0). This is the source of the position-generalization issue noted above.
• —Wallclock: ~14 hours / Bedrock spend: ~$30

DPO training

• —TRL 1.5.1 DPOTrainer with ref_model=None (PEFT adapter-disable trick saves 28 GB ref-model memory)
• —LoRA r=32, alpha=64, target=all-linear (same shape as Stage 1/2)
• —LR 5e-6 (10× lower than SFT), beta 0.1, sigmoid loss
• —37 steps × batch 32 = 1 epoch on 1167 pairs
• —Wallclock: ~6 minutes
• —Final metrics: rewards/margins 0.19 (from 0.01), rewards/accuracies 64% (from 36%), train_loss 0.666

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-14B",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
# Load DPO adapter directly (it absorbs the SFT base inside)
model = PeftModel.from_pretrained(base, "ianlee1996/pokerbench-qwen3-14b-lora-dpo")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-14B")

system_prompt = (
    "You are a specialist in playing 6-handed No Limit Texas Holdem. "
    "Output ONLY the optimal action with no explanation. "
    "Valid formats: 'fold', 'check', 'call', 'bet N', 'raise N', 'all-in'."
)
messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": "<paper-format game scenario...>"},
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=16, temperature=0.1, top_p=0.95, do_sample=True)
print(tokenizer.decode(out[0, inputs.input_ids.shape[1]:], skip_special_tokens=True))

Reproducibility

Pipeline + experiment configs at https://github.com/IanLiYi1996/PokerBench:

# 1. Self-play collection (~9 hours, ~$50 Bedrock)
.venv/bin/python -m scripts.collect_selfplay --config configs/eval/rl_selfplay_5k.yaml

# 2. Counterfactual extraction (~14 hours, ~$30 Bedrock)
.venv/bin/python -m scripts.extract_preferences \
    --hand-logs data/rl/selfplay_5k.jsonl \
    --out data/rl/preferences_5k.jsonl \
    --adapter checkpoints/06_qwen3_14b_lora_mixed_50_50/checkpoint-250 \
    --n-mc 10 --max-workers 8 \
    --bedrock-model deepseek.v3.2

# 3. DPO (~6 min)
.venv/bin/python -m scripts.train --config configs/experiments/07_qwen3_14b_dpo_vs_gptoss.yaml

The 1167 preference pairs and 5000 raw self-play hands used for training are public at `ianlee1996/pokerbench-rl-dpo`.

Honest reporting

This checkpoint is published as a research artifact, not a production-ready model. The headline takeaways:

• —DPO didn't break SFT: paper EM and PE AA both improved or stayed flat. The TRL DPOTrainer + PEFT adapter-disable trick is operationally sound.
• —DPO didn't beat SFT vs gpt-oss-120b in head-to-head: the +3 bb/100 target failed (delta −0.13 BB, statistical noise). Likely root cause: the counterfactual MC opponent (deepseek.v3.2) was different from the eval opponent (gpt-oss-120b), so DPO learned a "best response to deepseek", not a "best response to gpt-oss".
• —Position generalization is the bigger limitation: hero=UTG-only training caused CO position to regress by 25 BB/100. A successor checkpoint trained with hero=all-6-seats is the obvious fix.

Citation

@inproceedings{zhuang2025pokerbench,
  title={PokerBench: Training Large Language Models to become Professional Poker Players},
  author={Zhuang, Richard and Gupta, Akshat and Yang, Richard and Rahane, Aniket and Li, Zhengyu and Anumanchipalli, Gopala},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  year={2025},
  url={https://arxiv.org/abs/2501.08328}
}

License

Apache-2.0, matching Qwen/Qwen3-14B and the PokerBench dataset.