Datasetpublic

PersonaBias/bias_evaluation_sets

Persona Bias Evaluation Sets This dataset contains evaluation sets derived from full-model persona behavior. Each row is an original task sample grouped by whether changing the persona makes the model behavior biased, unbiased, or all-wrong. Repository Layout Hugging Face dataset config = model Hugging Face dataset split = validation or test Behavioral subset = eval_set column data/<model>/validation.jsonl.gz data/<model>/test.jsonl.gz manifest.jsonl… See the full description on the dataset page: https://huggingface.co/datasets/PersonaBias/bias_evaluation_sets.

sourceHugging Faceotherupdated 3mo agoView on Hugging Face

0likes39downloads

Dataset Card

Persona Bias Evaluation Sets

This dataset contains evaluation sets derived from full-model persona behavior. Each row is an original task sample grouped by whether changing the persona makes the model behavior biased, unbiased, or all-wrong.

Repository Layout

—Hugging Face dataset config = model
—Hugging Face dataset split = validation or test
—Behavioral subset = eval_set column

text

data/<model>/validation.jsonl.gz
data/<model>/test.jsonl.gz
manifest.jsonl

Evaluation Sets

eval_set	Meaning
`biased`	At least one persona value produces the correct answer and at least one persona value produces a wrong answer
`unbiased`	All persona values preserve the correct answer
`all_wrong`	All persona values produce wrong answers; this is saved as a separate diagnostic set

Row Counts

Model	Rows
`Llama-3.1-8B-Instruct`	68308
`Llama-3.2-1B-Instruct`	73584
`Llama-3.2-3B-Instruct`	73584
`Qwen2.5-7B-Instruct`	65316

Split	Rows
`test`	156280
`validation`	124512

Eval set	Rows
`all_wrong`	38031
`biased`	19154
`unbiased`	223607

Load Dataset

python

from datasets import load_dataset

ds = load_dataset(
    "PersonaBias/bias_evaluation_sets",
    "Llama-3.2-3B-Instruct",
    split="test",
)

print(ds)
print(ds[0])

Filter by Task, Axis, and Eval Set

python

task_axis_ds = ds.filter(
    lambda row: row["task"] == "arc_challenge"
    and row["axis"] == "emotion"
)

biased = task_axis_ds.filter(lambda row: row["eval_set"] == "biased")
unbiased = task_axis_ds.filter(lambda row: row["eval_set"] == "unbiased")
all_wrong = task_axis_ds.filter(lambda row: row["eval_set"] == "all_wrong")

print(len(biased), len(unbiased), len(all_wrong))

Shorter version:

python

biased_arc_emotion = ds.filter(
    lambda row: row["task"] == "arc_challenge"
    and row["axis"] == "emotion"
    and row["eval_set"] == "biased"
)

Important Schema Note

The source rows come from multiple tasks and persona axes. Some original fields have different types across tasks, for example numeric labels in one task and string labels in another. Some dictionary fields also have different keys across axes.

To make the Hugging Face dataset load reliably, source payload fields are stored as strings or JSON strings. The main filtering columns are ready to use directly. Some value columns are stored as strings for schema stability:

text

model
split
source_split
task
axis
eval_set
sample_id
task_label
groundtruth
sample

Structured fields are stored as JSON strings:

text

persona_results_json
predictions_by_persona_json
correctness_by_persona_json
unique_predictions_json
changed_persona_pairs_json
source_json

Parse Typed Fields

python

import json

def as_bool(x):
    return str(x).lower() == "true"

def as_int(x, default=None):
    text = str(x).strip()
    return int(text) if text else default

row = biased[0]

persona_results = json.loads(row["persona_results_json"])
predictions_by_persona = json.loads(row["predictions_by_persona_json"])
correctness_by_persona = json.loads(row["correctness_by_persona_json"])
source_row = json.loads(row["source_json"])

unique_prediction_count = as_int(row["unique_prediction_count"])
all_personas_wrong = as_bool(row["all_personas_wrong"])

Ready-to-Use Helper

python

import json

def as_bool(x):
    return str(x).lower() == "true"

def as_int(x, default=None):
    text = str(x).strip()
    return int(text) if text else default

def parse_eval_row(row):
    return {
        **row,
        "sample_id_int": as_int(row["sample_id"]),
        "unique_prediction_count_int": as_int(row["unique_prediction_count"]),
        "changed_persona_pair_count_int": as_int(row["changed_persona_pair_count"]),
        "all_personas_correct_bool": as_bool(row["all_personas_correct"]),
        "all_personas_wrong_bool": as_bool(row["all_personas_wrong"]),
        "mixed_correctness_bool": as_bool(row["mixed_correctness_across_personas"]),
        "prediction_changed_bool": as_bool(row["prediction_changed_across_personas"]),
        "persona_results": json.loads(row["persona_results_json"]),
        "predictions_by_persona": json.loads(row["predictions_by_persona_json"]),
        "correctness_by_persona": json.loads(row["correctness_by_persona_json"]),
        "unique_predictions": json.loads(row["unique_predictions_json"]),
        "changed_persona_pairs": json.loads(row["changed_persona_pairs_json"]),
        "source": json.loads(row["source_json"]),
    }

Example Analysis

python

from datasets import load_dataset

repo = "PersonaBias/bias_evaluation_sets"
model = "Llama-3.2-3B-Instruct"

ds = load_dataset(repo, model, split="test")

for task in sorted(set(ds["task"])):
    for axis in sorted(set(ds["axis"])):
        subset = ds.filter(lambda row: row["task"] == task and row["axis"] == axis)
        counts = {
            name: len(subset.filter(lambda row: row["eval_set"] == name))
            for name in ["biased", "unbiased", "all_wrong"]
        }
        print(task, axis, counts)

Available Tasks and Axes

Main tasks:

text

arc_easy
arc_challenge
ethics
safety
sst2

Axes:

text

emotion
gender
race
religion

Notes:

—sst2 has validation only because SST-2 test labels are unavailable.
—gsm8k scans are not included in this release.
—all_wrong is diagnostic and should not be merged into biased or unbiased.

Manifest

manifest.jsonl records each source file used to build the dataset, including model, task, split, axis, eval set, row count, and whether an expected source file was missing.