tintin1027/atomic-metrics-demographic-training-size

Atomic Metrics: Demographic Training-Size Analysis

Complete offline reproduction bundle for the effect of batch-selected training size on demographic preference prediction.

Version 2 — replaces the fixed-bank analysis. Select k extraction batches (five pairs each), use only their metrics and their 5k training pairs to refit BT/LR, then evaluate on cached test200 scores restricted to those metrics. Both the training rows and metric columns change with size. Extraction/refinement outputs and original scoring contexts are reused; this is not a fresh end-to-end extraction-and-rescoring run. No model API access or API keys are needed to reproduce the analysis.

Contents

• —24 configurations: four backbones × two tasks × three demographic factors.
• —Backbones: GPT-5.5, GPT-5.6-sol, Qwen3.8-max, Gemini-3.8-flash.
• —Tasks: Trouble Solution and Value Controversy; factors: locale, gender, age.
• —Each configuration contains the actual final metric bank, available raw extracted metrics, 100 labeled training score vectors, and 200 labeled test score vectors. Across configurations: 2,400 training score rows and 4,800 test score rows. The six underlying splits are shared across backbones, not independent datasets.
• —Source preference records and demographic-side inputs, preserving original sample IDs.
• —Original BT/LR fitted coefficients, CV records, features and test predictions.
• —All 2,880 learning-curve evaluation records, including batch IDs, subset sample IDs, metric IDs/counts, CV fold counts and selected regularization, plus summary CSV and source hashes.
• —Offline analysis code, pinned dependency versions, methodological notes, and PDF/PNG/SVG plots.

Download the complete bundle

from huggingface_hub import snapshot_download
snapshot_download(
    repo_id="tintin1027/atomic-metrics-demographic-training-size",
    repo_type="dataset",
    local_dir="atomic-metrics-demographic-training-size",
)

The companion reproduction_bundle.tar.gz contains the same release files (excluding itself and the outer checksum file). Downloading the snapshot includes both unpacked files and this convenience archive.

Reproduce

cd atomic-metrics-demographic-training-size
python -m venv .venv
.venv/bin/pip install -r requirements.txt
# Check exact batch/sample/metric mappings and reference endpoints:
.venv/bin/python scripts/validate_demographic_training_size.py
# Redraw the released curves without refitting:
.venv/bin/python scripts/analyze_demographic_training_size.py --plot-only
# Refit all subsets and regenerate figures (CPU only; no paid API calls):
.venv/bin/python scripts/analyze_demographic_training_size.py --repeats 10

The release was produced with Python 3.12. Before refitting, preserve outputs/demographic_training_size/ if you want to retain the released results separately: the script rewrites these outputs. It verifies that all 48 full-training accuracies reproduce the saved reference results.

Load scores as a Hugging Face dataset

from datasets import load_dataset
scores = load_dataset(
    "tintin1027/atomic-metrics-demographic-training-size",
    "gpt55_trouble_solution_locale",
)

Each configuration exposes train (100) and test (200). metric_values_a and metric_values_b map metric IDs to cached 0–10 scores; human_preference is A or B. The fitting feature is (A score − B score) / 10, in the metric order resolved by the provided loader. Metric definitions live alongside the scores in metric_bank.json. The score records also retain their original scoring protocol, model and bank-signature metadata.

File map

Analysis protocol and limitations

Sizes are 10, 20, 40, 60, 80 and 100 pairs, corresponding to 2, 4, 8, 12, 16 and 20 extraction batches. Ten uniformly shuffled batch orders use seeds 1700–1709, with nested prefixes across sizes. The same batches and sample IDs are used across backbones. Sampling is by batch, not individually label-stratified. Source batch membership partitions the original train100 into disjoint groups of five.

Each scored metric has its exact source batch and raw metric ID. All source banks use none_unique_raw_metric_id; we filter their metrics by source batch without introducing semantic merging or new metrics. Training matrices contain only selected rows AND columns. Test matrices retain all original 200 rows but use only selected metric columns.

Regularization uses training-subset-only stratified CV with min(5, minority-class count) folds, seed 17. All smallest sampled subsets have at least two examples per class. One Qwen / Value Controversy / Locale draw at n=10 (seed 1702, batches 001 and 004) has zero metrics. It is retained without resampling: the empty linear score is zero, probability 0.5, and the existing tie rule predicts A. No fit/CV occurs for that draw; parameter is null and cvfolds=0. Its accuracy is included in the plotted statistics and explicitly marked nometricszeromargintieA, not a successful fit. Test labels never select parameters or batch subsets. At n=100, the unchanged complete fit is reused across repetitions, giving zero subset SD. Bands are ±1 sample SD across batch subsets, not confidence intervals. Overview curves average the six settings within repetition first.

All train/test sample IDs are disjoint, but the original demographic vote split does not guarantee disjoint underlying prompts or response pairs. Metrics reuse existing batch extraction/refinement outputs. Cached scores retain the original scoring context, calibration and provider configurations, including any full-bank context. We do not claim to reproduce the effect of freshly scoring a smaller bank. The earlier release's fixed-full-bank weight-only curves are superseded by this version.

Data provenance and terms

Source preference records identify their upstream source in source_dataset; they derive from the project's Community Alignment demographic preference experiments. Original upstream terms continue to apply to source content and annotations. This bundle does not grant a new blanket license to third-party material. Model-derived metrics and scores are supplied for reproduction of the reported experiments. Pseudonymous source IDs and original provenance fields are retained for split auditing. This release contains no API credentials, private environment files or provider request logs.