nips234678/poolbench

PoolBench

PoolBench is a diagnostic benchmark for evaluating pooling strategies in decoder-only large language models. Every embedding pipeline implicitly chooses a pooling strategy — last token, mean pooling, attention-weighted, etc. — yet this choice is almost never studied systematically. PoolBench provides the first controlled, multi-concept, multi-model evaluation framework for this decision.

What is a pooling strategy?

A pooling strategy is a function that maps a sequence of per-token hidden states This vector is then used for classification, similarity search, or steering. PoolBench asks: which pooling strategy best captures a given semantic concept in a given model at a given layer?

Benchmark overview

The 17 Concepts

Concepts span five linguistic families, chosen to cover a range of signal density and linguistic depth:

Sensitive concepts: toxicity and depression are included in all metric computations but their steered text outputs are withheld from public release (available on request for research use).

The 19 Pooling Strategies

Evaluation Dimensions

D1 — Concept Separability (AUROC)

Linear probe AUROC measuring how well a pooling strategy separates positive from negative passages for each concept. Training: 5-fold OOF on 700 train passages per class. No labels from the test split are used for D1 training.

D2 — Steered Concept Prevalence (SCP)

Measures whether a pooling strategy's steering vector (DiffMean) actually steers generated text toward the target concept. A Classifier B (fine-tuned BERT) scores each steered generation. SCP = correlation between alpha (steering strength) and Classifier B score.

D3 — Disentanglement

Measures whether steering toward concept A avoids contaminating a linguistically-distant (LD) and linguistically-close (LC) neighbour concept.

$$\text{Disent}c = 1 - \frac{\DeltaB}{\Delta_A}$$

Where $\DeltaA$ = SCP on target concept, $\DeltaB$ = SCP on neighbour concept under the same steering vector.

Dataset Structure

poolbench/
  {concept}/
    train_pos.jsonl    # 700 positive training passages
    train_neg.jsonl    # 700 negative training passages
    test_pos.jsonl     # 300 positive test passages
    test_neg.jsonl     # 300 negative test passages

Each JSONL record:

{
  "id": "academic_tone_train_pos_0001",
  "text": "...",
  "label": 1,
  "domain": "academic",
  "token_count": 347,
  "matched_pair_id": null,
  "split": "train"
}

Fields:

• —id — unique identifier with concept, split, class, and index
• —text — passage text (300–500 LLaMA-3.1-8B tokens except toxicity/deference)
• —label — 1 = positive, 0 = negative
• —domain — source domain (e.g., academic, news, social, legal_us)
• —token_count — exact token count under LLaMA-3.1-8B tokenizer
• —matched_pair_id — non-null for matched-pair concepts (negative is a controlled rewrite of the positive)
• —split — train or test

Corpus Construction

Positive and negative passages were constructed under strict controls:

• —700/700 train, 300/300 test per concept — balanced classes to avoid probe bias
• —300–500 token window — enforced by LLaMA-3.1-8B BPE tokenizer at build time
• —≥3 source domains per concept — prevents domain vocabulary from confounding pooling strategy comparisons
• —±25 token matching rule — for matched-pair concepts, the positive and its negative rewrite differ by ≤25 tokens
• —MD5-based deduplication — within-class, cross-class, and train/test leak checks
• —Seed-word contamination filter — negatives must not contain the concept's seed words

See the GitHub repository for full construction notes and per-concept source tables.

Source datasets

Quick Start

from datasets import load_dataset

# Load one concept
ds = load_dataset("nips234678/poolbench", data_dir="academic_tone")

# Load all concepts
from pathlib import Path
concepts = [
    "academic_tone", "bureaucratic", "causation", "code_docs",
    "conditionality", "contrast", "deference", "depression",
    "frustration", "hedging", "imdb_sentiment", "legal_formality",
    "narrative", "negation_density", "numerical_precision",
    "planning", "toxicity",
]
for concept in concepts:
    ds = load_dataset("nips234678/poolbench", data_dir=concept)

To reproduce the full pipeline from activations to leaderboard:

git clone https://github.com/2023mc21517-arch/poolbench.git
cd poolbench
pip install -e ".[dev]"

# Download corpus
huggingface-cli download nips234678/poolbench --repo-type dataset --local-dir data/corpora

# Run full pipeline (requires GPU, ~8h per model on A100)
python scripts/run_model.py --model mistral7b --device cuda:0

Related Artifacts

Ethical Considerations

• —Toxicity and depression concepts include passages containing toxic language and expressions of psychological distress, respectively. These are sourced from publicly available datasets with established research use. Steered text outputs for these two concepts are withheld from public release.
• —No personal information — all passages are sourced from public corpora with no PII.
• —Model outputs — steered generation outputs are research artifacts demonstrating steering vector magnitude effects; they are not endorsements of the content.
• —Intended use — evaluation of pooling strategies in language model representations. Not intended for clinical use, content moderation in production, or as training data for generation models.

Citation

@dataset{poolbench2026,
  title        = {{PoolBench}: A Benchmark for Pooling Strategies in Decoder-Only Language Models},
  author       = {Anonymous},
  year         = {2026},
  publisher    = {HuggingFace},
  url          = {https://huggingface.co/datasets/nips234678/poolbench},
}

Citation will be updated with full author list and DOI upon paper acceptance.