evaleval/social_impact_eval_annotations

1---2language:3- en4license: odc-by5size_categories:6- 1K<n<10K7dataset_info:8  features:9  - name: provider10    dtype: string11  - name: name12    dtype: string13  - name: size14    dtype: string15  - name: variant16    dtype: string17  - name: version18    dtype: string19  - name: sector20    dtype: string21  - name: openness22    dtype: string23  - name: region24    dtype: string25  - name: country26    dtype: string27  - name: source_id28    dtype: string29  - name: is_first_party30    dtype: bool31  - name: category32    dtype: int6433  - name: year34    dtype: int6435  - name: metadata36    dtype: string37  - name: score38    dtype: float6439  - name: is_model_release40    dtype: bool41  splits:42  - name: train43    num_bytes: 118048144    num_examples: 424145  download_size: 5929246  dataset_size: 118048147configs:48- config_name: default49  data_files:50  - split: train51    path: data/train-*52---53# Dataset Card for social_impact_eval_annotations54The `social_impact_eval_annotations` dataset contains annotations for first-party and third-party social impact evaluation reporting practices for 186 models along seven dimensions.55## Dataset Details56### Dataset Description57The `social_impact_eval_annotations` dataset comprises analyzed social impact evaluation reporting for 186 foundation models released between 2018-2025. Each model's reporting is evaluated across seven social impact dimensions: bias and representational harms, sensitive content, disparate performance, environmental costs and emissions, privacy and data protection, financial costs, and data/content moderation labor. The reporting is scored on a 0-3 scale to indicate the depth and clarity of reported evaluations. 58 59- **Curated by:** EvalEval Coalition60- **Shared by:** EvalEval Coalition61- **Language(s) (NLP):** English62- **License:** Open Data Commons Attribution License (ODC-By)63 64### Dataset Sources65- **Repository:** https://github.com/evaleval/social_impact_eval_annotations_code66- **Paper:** https://arxiv.org/pdf/2511.0561367 68## Uses 69### Direct Use70This dataset is intended for:71- Analyzing social impact evaluation reporting72- Informing the development of evaluation standards and reporting frameworks 73 74### Out-of-Scope Use75This dataset should not be used for:76- Assessing actual model societal impact or deployment suitability – scores reflect reporting presence and detail, not the quality or adequacy of evaluations themselves77 78## Dataset Structure79 80Each row represents one evaluation instance, capturing the level of reporting detail given for a specific model that was evaluated on one social impact category in one source, e.g., paper, leaderboard, blog. A single model can have multiple rows (one per evaluation category per source).81 82### Data Fields83 84* `provider`: Organization that developed the model (str)85* `name`: Base model name (str)86* `size`: Model parameter count when available (str)87* `variant`: Model variant specification (str)88* `version`: Specific model version or release identifier (str)89* `sector`: Organization sector (str)90* `openness`: Model weight accessibility (str)91* `region`: Provider headquarters region (str)92* `country`: Provider headquarters country (str)93* `source_id`: Unique identifier for the source of the evaluation report (str)94* `is_first_party`: Whether reported evaluation was conducted by the model provider (bool)95* `category`: Social impact category identifier (int, 1-7) corresponding to the seven dimensions96* `year`: Year of report (int)97* `metadata`: Metadata including URLs, full release dates, and other source information (dict)98* `score`: The level of reporting detail of the evaluation, scored on 0-3 scale (float)99* `is_model_release`: Whether instance is from model release-time reporting (bool)100 101 102## Dataset Creation103### Curation Rationale104As foundation models become central to high-stakes AI systems, governance frameworks increasingly rely on evaluations to assess risks and capabilities. While general capability evaluations are common, social impact assessments remain fragmented, inconsistent, or absent. 105 106This dataset was created to move beyond anecdotal evidence and provide systematic documentation of how model developers and the research community evaluate and report on societal impacts of AI systems.107 108 109### Source Data110#### Data Collection and Processing111For details, please see Section 3 in our paper.112 113We first compiled a list of models by triangulating across public sources (e.g., FMTI, LMArena). Next, we expanded this list with providers referenced in leaderboards and technical reports. We selected all official model releases, including those fine-tuned by the original developer but excluding community fine-tuned versions. For multimodal models, we include those architecturally distinct systems that are recognized as foundation models in the literature or have widespread adoption by the research community. We disambiguate consumer-facing applications (e.g., ChatGPT) to the underlying model where possible and skip it otherwise. 114 115For these models, we identified sources for first-party and third-party reports through complementary searches:116- **First-party**: Manual search of provider websites for papers, technical reports, model cards, system cards, blogs, and press releases117- **Third-party**: Automatic search using Paperfinder for peer-reviewed academic papers118- **Leaderboards**: Targeted queries on Google Search and Hugging Face Spaces119 120#### Who are the source data producers?1211. First-party developers: Foundation model developers from industry, academia, government, and non-profit organizations.1222. Third-party evaluators: Independent researchers, academic institutions, and evaluation organizations reporting conducted social impact evaluations on released models.123 124#### Annotation process125 126In total, we compiled data from 186 first-party release time sources and 248 post-release sources (out of which 211 are fully third-party, 17 are fully first-party, and there are 20 sources by model providers that report both results for their own model (labeled as first-party) and those of other providers’ (labeled as third-party)). 127 128This forms 4241 evaluation instances. Each instance was annotated against the seven social impact dimensions using a standardized guide. Annotations were performed by individual researchers, with manual spot checks for consistency.129 130The social impact categories are:1311. Bias, Stereotypes, and Representational Harms1322. Cultural Values and Sensitive Content1333. Disparate Performance1344. Environmental Costs and Carbon Emissions1355. Privacy and Data Protection1366. Financial Costs1377. Data and Content Moderation Labor138 139The scoring criteria are:140- **0**: No mention of the category, or only generic references without evaluation details.141- **1**: Vague mention of evaluation (e.g., “We check for X” or “Our model can exhibit X”).142- **2**: Evaluation described with concrete information about methods or results (e.g., “Our model scores X% on the Y benchmark”) but lacking methodological detail.143- **3**: Evaluation methods described in sufficient detail to enable meaningful understanding and/or reproduction. Where applicable, the study design is documented (dataset, metric, experiment design, annotators), and results are contextualized with assumptions, limitations, and practical implications.144 145For cost-related categories (environmental and financial), we applied slightly modified criteria to account for reporting based on hardware specifications or resource usage rather than benchmark-style evaluations:146- **0**: No reporting.147- **1**: Same as above, or when reported technical details (e.g., FLOPs, GPU type, runtime) could indirectly be used to estimate costs.148- **2**: Concrete values reported for a non-trivial part of model development or hosting, but derivation method unclear.149- **3**: Concrete values reported together with contextual details and the derivation method.150 151For financial costs, we excluded first-party customer-facing pricing from consideration, as it reflects product strategy rather than system costs. Third-party cost estimates for completing specific tasks were included.152 153#### Who are the annotators?154Researchers from the EvalEval Coalition created the annotations.155 156#### Personal and Sensitive Information157The dataset contains no personal information about individuals. All data sources are publicly available documents (technical reports, academic papers, model cards, etc.).158 159## Bias, Risks, and Limitations160This dataset may overrepresent models from prominent providers and English sources.161 162Our scoring captures reporting presence and specificity, but does not reflect methodological soundness, depth, or coverage of evaluations. Missing instances in this dataset may stem from limitations in our search approach or reflect reporting gaps, rather than evaluation gaps in practice. 163 164### Recommendations165Analyses should consider potential overrepresentation of prominent providers and English sources. 166Scores should be interpreted as perceived quality of reporting practices rather than actual model societal impact or capabilities.167 168## Citation169**BibTeX:**170```bibtex171@misc{reuel2025social,172    title={Who Evaluates AI's Social Impacts? Mapping Coverage and Gaps in First and Third Party Evaluations},173    author={Anka Reuel and Avijit Ghosh and Jenny Chim and Andrew Tran and Yanan Long and Jennifer Mickel and Usman Gohar and Srishti Yadav and Pawan Sasanka Ammanamanchi and Mowafak Allaham and Hossein A. Rahmani and Mubashara Akhtar and Felix Friedrich and Robert Scholz and Michael Alexander Riegler and Jan Batzner and Eliya Habba and Arushi Saxena and Anastassia Kornilova and Kevin Wei and Prajna Soni and Yohan Mathew and Kevin Klyman and Jeba Sania and Subramanyam Sahoo and Olivia Beyer Bruvik and Pouya Sadeghi and Sujata Goswami and Angelina Wang and Yacine Jernite and Zeerak Talat and Stella Biderman and Mykel Kochenderfer and Sanmi Koyejo and Irene Solaiman},174    year={2025},175    eprint={2511.05613},176    archivePrefix={arXiv},177    primaryClass={cs.CY},178    url={https://arxiv.org/abs/2511.05613},179    note={Preprint}180}181```182**APA:**183> Reuel, A., Ghosh, A., Chim, J., Tran, A., Long, Y., Mickel, J., Gohar, U., Yadav, S., Ammanamanchi, P. S., Allaham, M., Rahmani, H. A., Akhtar, M., Friedrich, F., Scholz, R., Riegler, M. A., Batzner, J., Habba, E., Saxena, A., Kornilova, A., Wei, K., Soni, P., Mathew, Y., Klyman, K., Sania, J., Sahoo, S., Bruvik, O. B., Sadeghi, P., Goswami, S., Wang, A., Jernite, Y., Talat, Z., Biderman, S., Kochenderfer, M., Koyejo, S., & Solaiman, I. (2025). Who evaluates AI's social impacts? Mapping coverage and gaps in first and third party evaluations (arXiv:2511.05613). arXiv. https://arxiv.org/abs/2511.05613184 185## Dataset Card Authors186[Jenny Chim](mailto:c.chim@qmul.ac.uk)187 188## Dataset Card Contact189[Anka Reuel](mailto:anka.reuel@stanford.edu), [Avijit Ghosh](mailto:avijit@huggingface.co), [Jenny Chim](mailto:c.chim@qmul.ac.uk)