CoolFace
Datasetpublic

Aeryx-ai/aae-dialect-fairness

AAE Dialect-Fairness Set A reusable set for debiasing hate/toxicity classifiers against African-American English (AAE) false positives. Off-the-shelf classifiers flag benign AAE text as toxic at 2x+ the rate of benign General-American English (Sap et al. 2019). This set provides (1) high-AAE benign text to augment training so a model can't use dialect as a toxicity cue, and (2) a held-out dialect-balanced benchmark to measure the residual gap. Built for the… See the full description on the dataset page: https://huggingface.co/datasets/Aeryx-ai/aae-dialect-fairness.

sourceHugging Faceotherupdated 2mo agoView on Hugging Face
1likes25downloads
Dataset Card

AAE Dialect-Fairness Set

A reusable set for debiasing hate/toxicity classifiers against African-American English (AAE) false positives. Off-the-shelf classifiers flag benign AAE text as toxic at 2x+ the rate of benign General-American English (Sap et al. 2019). This set provides (1) high-AAE benign text to augment training so a model can't use dialect as a toxicity cue, and (2) a held-out dialect-balanced benchmark to measure the residual gap.

Built for the guardian-dialect-fair-hate classifier, where adding aae_safe was the lever that closed the dialect FP gap (recall@parity 0.81 → 0.95).

Configs

configrowswhat
aae_safe30,946High-AAE benign text (label = not-hate). Training augmentation. Fields: text, aae_score, source.
dialect_fp_eval3,000Dialect-balanced benign benchmark (1.5k high-AAE + 1.5k low-AAE). Score a classifier on it; the FP rate should be equal across bucket. Fields: text, aae_score, source, bucket.

aae_score is the African-American topic proportion from TwitterAAE (Blodgett et al. 2016); high-AAE is >= 0.5.

How to use (debiasing)

Mix aae_safe into your negatives (label = not-toxic) so high-AAE text is well-represented in the safe class, then gate release on dialect_fp_eval: the false-positive rate on bucket=high_aae must match bucket=low_aae within tolerance, not just overall F1.

Provenance & caveats

  • —Derived, not original. Text is a filtered + dialect-scored subset of public datasets, primarily `LabHC/moji` (~90%; the DeepMoji race dataset, Twitter-derived), plus Measuring Hate Speech, Civil Comments, DynaHate, ToxiGen. Licensing/ToS inherit from those sources — review before any public redistribution.
  • —TwitterAAE is a Twitter-domain distant-supervision dialect proxy (geography→demographics), not ground truth and noisier off-domain. It is a language proxy, not a race label.
  • —Benign labels for moji come from its sentiment task (not a hate annotation); a small number of false negatives are possible.
  • —English only.

Citation references

Sap et al. 2019 (Risk of Racial Bias in Hate Speech Detection); Blodgett et al. 2016 (TwitterAAE); Elazar & Goldberg 2018 (moji / demographic text); Kennedy et al. (Measuring Hate Speech).