Datasetpublic

ianlee1996/pokerbench-rl-dpo

PokerBench RL — Counterfactual DPO Preference Data DPO preference pairs and raw self-play logs for training a Texas Hold'em LLM to exploit non-GTO opponents, addressing the PokerBench paper's Future Work observation that pure SFT models lose to GPT-4-style "donking" strategies. This dataset feeds the ianlee1996/pokerbench-qwen3-14b-lora-dpo checkpoint training. How it was built Self-play (5000 hands): ianlee1996/pokerbench-qwen3-14b-lora-mixed (Qwen3-14B + LoRA… See the full description on the dataset page: https://huggingface.co/datasets/ianlee1996/pokerbench-rl-dpo.

sourceHugging Faceapache-2.0updated 3mo agoView on Hugging Face

0likes51downloads

Dataset Card

PokerBench RL — Counterfactual DPO Preference Data

DPO preference pairs and raw self-play logs for training a Texas Hold'em LLM to exploit non-GTO opponents, addressing the PokerBench paper's Future Work observation that pure SFT models lose to GPT-4-style "donking" strategies.

This dataset feeds the `ianlee1996/pokerbench-qwen3-14b-lora-dpo` checkpoint training.

How it was built

1.Self-play (5000 hands): `ianlee1996/pokerbench-qwen3-14b-lora-mixed` (Qwen3-14B + LoRA, GTO-trained) plays as Team A (3 seats) against openai.gpt-oss-120b-1:0 on AWS Bedrock as Team B (3 seats). Captured every decision (state, action, available moves, prompt) per hand into selfplay_5k.jsonl.

1.Counterfactual EV extraction: For each hero hand, pick one key decision (river > turn > flop > preflop ≥ 2BB), generate 2 candidate actions (low-temperature + high-temperature, with structural fallback), and run N=10 monte-carlo rollouts per candidate with deepseek.v3.2 driving every other seat to showdown. The action with higher EV becomes chosen; the other becomes rejected. Pairs with |EV_chosen - EV_rejected| < 0.5 BB (noise floor) are dropped.

1.DPO: feed the resulting 1167 preference pairs to TRL DPOTrainer on top of the SFT adapter.

Files

`preferences_5k.jsonl` — 1167 DPO preference pairs (1.3 MB)

Each line is one record:

json

{
  "prompt": "<full paper-format game scenario string>",
  "chosen": "raise 2.0",
  "rejected": "fold",
  "ev_chosen_bb": 4.30,
  "ev_rejected_bb": 0.00,
  "ev_diff_bb": 4.30,
  "n_mc_chosen": 10,
  "n_mc_rejected": 10,
  "metadata": {
    "hand_id": 16,
    "street": "preflop",
    "key_decision_rule": "D",
    "hero_position": "UTG"
  }
}

Quality stats:

Metric	Value
Total pairs	1167
EV gap median	2.10 BB
EV gap p25 / p75	1.35 / 4.76 BB
EV gap max	63.95 BB (all-in spots)
EV gap min	0.50 BB (noise floor)
All > 0.5 BB threshold	✅

Decision-rule mix:

Rule	Street	Count	Pct
D	preflop (raise ≥ 2 BB)	998	85.5%
A	river (last decision)	113	9.7%
C	flop (last decision)	39	3.3%
B	turn (last decision)	17	1.5%

Preflop dominates because GTO opens at UTG mean ~85% of hands fold preflop and never reach a postflop decision.

`selfplay_5k.jsonl` — 5000 hands of raw self-play (35 MB)

Each line is one full hand log:

json

{
  "hand_idx": 7,
  "decisions": [
    {
      "seat_idx_paper": 0,
      "street": "preflop",
      "action": "raise 2.0",
      "pot_before": 1.5,
      "hero_invested_this_street": 0.0,
      "available_moves": ["fold", "call", "bet", "raise", "all-in"],
      "prompt_text": "<full paper-format scenario string>"
    },
    ...
  ],
  "starting_stacks": [100.0, 100.0, 100.0, 100.0, 100.0, 100.0],
  "final_stacks": [101.5, 99.5, 100.0, 100.0, 100.0, 99.0],
  "board_cards": ["Th", "3s", "2d", "5d", "Kh"],
  "hole_cards": {
    "0": ["Ac", "Ad"],
    "1": ["Kc", "9d"],
    ...
  },
  "seat_a_role": {
    "0": "A", "1": "B", "2": "A", "3": "B", "4": "A", "5": "B"
  }
}

Headline stats:

—5000 hands completed (no hand failures)
—Team A (ckpt-mixed) +46.10 bb/100 vs Team B (gpt-oss-120b)
—Reproduces the Stage 3 baseline (+46.6 bb/100) within margin of error
—Average 6.4 decisions per hand

Loading

python

from datasets import load_dataset

# DPO preference pairs (default; for trl DPOTrainer)
prefs = load_dataset("ianlee1996/pokerbench-rl-dpo", "preferences", split="train")
print(prefs[0])

# Raw self-play logs (for re-running counterfactual extraction or other RL approaches)
hands = load_dataset("ianlee1996/pokerbench-rl-dpo", "selfplay", split="train")
print(hands[0])

Reproduction

Full pipeline + experiment configs at https://github.com/IanLiYi1996/PokerBench:

bash

# 1. Self-play (~10 hours, ~$50 Bedrock)
.venv/bin/python -m scripts.collect_selfplay --config configs/eval/rl_selfplay_5k.yaml

# 2. Counterfactual extraction (~10 hours, ~$30 Bedrock)
.venv/bin/python -m scripts.extract_preferences \
    --hand-logs data/rl/selfplay_5k.jsonl \
    --out data/rl/preferences_5k.jsonl \
    --adapter checkpoints/06_qwen3_14b_lora_mixed_50_50/checkpoint-250 \
    --n-mc 10 --max-workers 8 \
    --bedrock-model deepseek.v3.2

# 3. DPO (~30 min)
.venv/bin/python -m scripts.train --config configs/experiments/07_qwen3_14b_dpo_vs_gptoss.yaml

Citation

If you use this dataset, cite the original paper:

bibtex

@inproceedings{zhuang2025pokerbench,
  title={PokerBench: Training Large Language Models to become Professional Poker Players},
  author={Zhuang, Richard and Gupta, Akshat and Yang, Richard and Rahane, Aniket and Li, Zhengyu and Anumanchipalli, Gopala},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  year={2025},
  url={https://arxiv.org/abs/2501.08328}
}

License

Apache-2.0, matching the PokerBench dataset license.