alisa-yingjia-wan/gapa

GAPA: Gender Associations of Physical Attributes

   

The GAPA dataset contains 14,706 human ratings of how strongly English physical descriptions (n=316, e.g., "a defined jawline", "a soft voice", "broad shoulders") are associated with a woman, a man, or a non-binary person.

_{Paper Figure 1 — an excerpt of the most gender-distinctive attributes in each ranking pattern, with their per-gender association ratings.}

How to use

from datasets import load_dataset, concatenate_datasets

gapa = load_dataset("alisa-yingjia-wan/gapa")          # cleaned, the default
gapa["human"]                                          # just the human-written attributes

everything = concatenate_datasets(list(gapa.values()))  # all 14,706 ratings
raw = load_dataset("alisa-yingjia-wan/gapa", "raw")    # before filtering

The code repository holds everything built on top of these ratings, including dataset analysis, LLM evaluation, LoRA training on GAPA, etc. python data/download_ratings.py there fetches this dataset into a working data/.

Background

Guidance in AI fairness and accessibility often recommends describing people by observable physical attributes rather than inferred identity labels, on the reasoning that describing what is visible avoids imposing identity claims. GAPA exists to test that premise, and finds it does not hold: these descriptions carry systematic, graded gender associations.

What is in it

315 physical attributes, each rated against all three genders. Every row is one rating by one annotator.

• —Which configuration you load says how filtered the ratings are: | Config | Rows | Description | |---|---|---| | clean (default) | 14,706 | The canonical table. Every published number is computed from this. | | raw | 16,002 | The same ratings before filtering, for auditing the cleaning. |

Clearning operates two filers, both applied at the level of a whole attribute:

1. 1.Incomplete attributes are dropped. An attribute must have ratings for all three of woman / man / nonbinary person; otherwise it cannot enter a gender comparison.
2. 2.Attention-check attributes are dropped. Six items were used purely as attention checks and are not real data: a balding crown, a bushy beard, a muscular neck, an androgynous build, an hourglass figure, curled eyelashes.

• —Which split you take says where the attributes came from. Check the paper for full details. | Split | Attributes | Rows (clean) | How they were produced | |---|---|---|---| | llm | 170 | 8,406 | LLM-generated candidate descriptions | | human | 50 | 2,300 | Written by people | | novel | 96 | 4,000 | Extracted from published novels |

The merged data from three sources sum to n=315 attributes after deduplication (one attribute, deep-set eyes, was independently produced by two of the sources and rated in both). Every row also keeps a source column, so a concatenated frame stays self-describing.

Column descriptions

Attention checks and pilot attributes

• —Attention checks are trials whose answer is obvious, included so that a disengaged annotator can be spotted: a bushy beard asked about a man, curled eyelashes asked about a woman. Each annotator saw five. The attention_check column marks whether a given trial was one, and attention_passed / n_attchecks_seen / n_attchecks_passed record how the annotator did. The six attributes that existed only to serve as checks are removed from clean by rule 2 above. Annotators, however, are kept: check performance is recorded but was never used to filter the sample, so every annotator's ratings are present by default. Filter on n_attchecks_passed if your analysis wants a stricter sample. <!-- For reference, 23 of 46 annotators in the human source and 47 of 80 in novel missed at least one of their five. -->

• —Pilot attributes are four items used as checks during the pilot round only, which were also shown as ordinary rated trials: a rugged jawline, a curvy build, painted nails, and prominent shoulders. Two of them are reverse-coded — the expected answer is a low rating rather than a high one (a rugged jawline asked about a woman, a curvy build asked about a man).

Because these attributes carry genuine ratings alongside their check trials, removing them wholesale would discard real data, so they are kept in `clean` by default. Three of the four survive (prominent shoulders was already removed by rule 1), bringing 192 check trials with them. Those trials are all marked attention_check == True:

  gapa = concatenate_datasets(list(load_dataset("alisa-yingjia-wan/gapa").values())).to_pandas()
  ratings_only = gapa[~gapa.attention_check]   # drops the 192 pilot check trials

Privacy

Annotators were recruited on Prolific under IRB-approved procedures. Prolific-issued identifiers were replaced with sequential pseudonyms (P001…, S001…) rather than hashes, so no amount of brute force recovers the original: a sequential label carries none of the original's information. <!-- The re-link table is held by the authors and is not distributed. Free-text comments and several demographic fields were dropped outright; age is retained by the authors' decision, as it is used in the paper's analysis. -->

Citation

If you use GAPA, please cite:

@inproceedings{wan2026gapa,
  title         = {How Humans and LLMs Read Gender into ``Gender-Neutral'' Physical Descriptions},
  author        = {Wan, Yingjia and Lin, Lin L. and Kreiss, Elisa},
  booktitle     = {Conference on Language Modeling (COLM)},
  year          = {2026},
  eprint        = {2609.16366},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2609.16366}
}