latenttensor26/racial-bias-llm

Color-Blind Bias in Frontier Language Models

This repository contains the data, code, and metadata accompanying the paper "Color-Blind Bias in Frontier Language Models: An Evaluation Framework with Validated Racial Attitude Scales", under double-blind review at NeurIPS 2026 (Evaluations & Datasets Track).

Submission status: anonymous for review.

What this evaluation measures

We applied six validated human racial-attitude scales to 32 frontier language models from 8 providers under three prompting conditions, yielding 302,354 logged item-level responses. The framework measures three analytical layers of racial bias:

• —Layer A — Explicit bias: ATB (Brigham, 1993), Bogardus Social Distance (Bogardus, 1925), OFR (Swim et al., 1995)
• —Layer B — Modern covert racism: MRS (McConahay, 1986), Racial Resentment Scale (Kinder & Sanders, 1996)
• —Layer C — Structural racial denial: CoBRAS (Neville et al., 2000)

Scores are standardized using Percent of Maximum Possible (POMP, 0–100; higher = stronger bias-consistent response patterns).

Repository structure

repo/
├── README.md                       This file
├── LICENSE                         MIT (code) + CC-BY 4.0 (data)
├── Croissant_RAI.json                  Croissant 1.0 metadata with RAI fields
├── SCALE_PROVENANCE.md             Provenance and licensing of the 6 scales
├── data/
│   ├── raw_responses.zip           302,354 logged item-level responses (compressed; ~224 MB uncompressed CSV)
│   ├── item_level_aggregated.csv   Per-item POMP-standardized scores
│   ├── scale_level_bias_scores.csv Per-scale bootstrap means and CIs
│   └── leaderboards/
│       ├── TABLE_LEADERBOARD_neutral.csv
│       ├── TABLE_LEADERBOARD_social_worker.csv
│       └── TABLE_LEADERBOARD_racist.csv
└── notebooks/
       ├── Bias_Racial_Code_All_Models.ipynb              Data collection (API calls across 32 models × 3 personas × 63 items × 50 iterations)
       └── LLM_Racial_Bias_Psychometric_Analysis.ipynb    Analysis pipeline (reproduces all tables and figures in the paper)

Reproduction instructions

1. Environment

The analysis was executed in the following environment (fingerprinted at the top of the master notebook):

• —Python 3.11.15
• —NumPy 2.1.3
• —Pandas 2.3.3
• —SciPy 1.16.3
• —statsmodels 0.14.6
• —scikit-learn 1.7.2
• —matplotlib 3.10.8
• —seaborn 0.13.2
• —Random seed: 42

2. Steps

# Clone the anonymous repository
# (URL provided in OpenReview submission)

# Install dependencies
pip install -r requirements.txt

# Open the notebook
jupyter notebook notebooks/LLM_Racial_Bias_Psychometric_Analysis.ipynb

The notebook reproduces all leaderboard tables, hypothesis-test outputs (H1–H5), bootstrap confidence intervals, principal component analysis, Generalizability-Theory variance components, and refusal-sensitivity analyses reported in the paper.

3. Compute requirements

• —The data-collection step (32 models × 3 prompts × 63 items × 50 iterations = 302,400 API calls) was completed prior to release. Re-running it is not required for analysis reproduction.
• —The downstream analysis runs on a standard laptop in approximately 45 minutes. No GPU required.

What is and is not in this release

Released

• —LLM responses (numeric responses + reasoning text) for all 32 models under all three prompting conditions
• —Per-item POMP-standardized scores with reverse-coding applied
• —Scale-level and layer-level aggregations
• —Leaderboard tables with bootstrap confidence intervals
• —Hypothesis-test outputs (H1–H5)
• —Robustness checks: MNAR triple-imputation, reverse-coding integrity, Generalizability-Theory variance components, PCA outputs
• —Executable notebooks: data collection + analysis pipeline (Jupyter)
• —Croissant 1.0 metadata with Responsible AI (RAI) fields documenting data collection, intended uses, biases, and limitations
• —Scale identifiers, item identifiers, response bounds, reverse-coding flags, and scoring rules for all six instruments

NOT released

• —Verbatim item text for the six psychometric instruments. These are copyrighted by the original authors/publishers and are not relicensed by this release. See SCALE_PROVENANCE.md for details.
• —Researchers wishing to replicate item-level scoring must obtain item text directly from the original publications, from APA PsycTESTS, or by contacting the original instrument authors.

Licenses

• —Code (notebooks, scripts, analysis pipeline): MIT License
• —Data (LLM responses, derived scores, aggregate tables, metadata): CC-BY 4.0
• —Third-party instrument item text: NOT relicensed — see SCALE_PROVENANCE.md

See LICENSE for full text.

Citation

Citation will be provided post de-anonymization. For now, please cite as:

[Author names redacted for anonymous review]. (2026). Color-Blind Bias in
Frontier Language Models: An Evaluation Framework with Validated Racial
Attitude Scales. Submitted to NeurIPS 2026 Evaluations & Datasets Track.

Contact

This repository is anonymous for double-blind review. Contact information will be provided post de-anonymization.

For now, questions can be submitted via the OpenReview discussion thread for the paper.