Datasetpublic

CGIAR/ifpri-ai-documents-markdown

GAIA / GARDIAN-CIGI Agricultural Research Corpus This dataset contains 21,726 agricultural research documents extracted from the GARDIAN repository and processed through the CIGI pipeline. Dataset Overview Property Value Total Documents 21,726 Total Size 623.27 MB Total Tokens 85,359,442 Total Pages 0 Languages 25 Unique Keywords 7,127 Resource Types 20 Date Generated 2026-07-31 02:55:20 Language Distribution… See the full description on the dataset page: https://huggingface.co/datasets/CGIAR/ifpri-ai-documents-markdown.

sourceHugging Facecc-by-4.0updated 2mo agoView on Hugging Face

1likes798downloads

Dataset Card

GAIA / GARDIAN-CIGI Agricultural Research Corpus

This dataset contains 21,726 agricultural research documents extracted from the GARDIAN repository and processed through the CIGI pipeline.

Dataset Overview

Property	Value
Total Documents	21,726
Total Size	623.27 MB
Total Tokens	85,359,442
Total Pages	0
Languages	25
Unique Keywords	7,127
Resource Types	20
Date Generated	2026-07-31 02:55:20

Language Distribution

Language	Count	Percentage
English	18921	87.1%
Hindi	1220	5.6%
French	534	2.5%
Spanish	456	2.1%
Arabic	123	0.6%
Portuguese	109	0.5%
Chinese	83	0.4%
Burmese	77	0.4%
Russian	53	0.2%
German	40	0.2%
Tajik	38	0.2%
Amharic	20	0.1%
Japanese	15	0.1%
Korean	10	0.0%
Italian	8	0.0%
Aragonese	6	0.0%
Bengali	3	0.0%
Yoruba	2	0.0%
Bosnian	2	0.0%
Vietnamese	1	0.0%
Swahili	1	0.0%
Kinyarwanda	1	0.0%
Danish	1	0.0%
Belarusian	1	0.0%
Basque	1	0.0%

Top Keywords

science, agriculture, academic, grobid, rag, cgiar, multilingual, documents, research, food-security, climate-adaptation, nutrition, food security, health, malnutrition, households, poverty, gender, food, developing countries, climate change, child nutrition, agricultural policies, maternal and child health, women

Resource Types

Type	Count
Brief	5888
Scientific Publication	4777
Working Paper	4099
Poster / Presentation	2157
Dataset / Tabular	1683
Report	1426
Book / Monograph	681
Factsheet	437
Other	230
Training Material	132
Online Sources	111
Magazine or Press item	34
Dataset / Geospatial	20
Multimedia	14
Report / Case study	11
Report / Financial	11
Report / Annual	7
Periodical / Journal	3
Map	3
Periodical / Magazine	2

Publication Years

2022 (3110), 2021 (1392), 2014 (1275), 2023 (1220), 2020 (1184), 2019 (1183), 2024 (1081), 2015 (979), 2018 (967), 2016 (935)

Rights

Per-document licensing varies. Distribution of metadata.rights:

Rights	Docs	Share
N/A	14042	64.6%
CC-BY	4853	22.3%
CP	1016	4.7%
CC-BY-NC-ND	728	3.4%
CC-BY-NC	727	3.3%
CC-BY-NC-SA	148	0.7%
CC0	86	0.4%
CC-BY-SA	61	0.3%
CP-NC	35	0.2%
CC-BY-ND	28	0.1%
CP-EDU	2	0.0%

Rights are heterogeneous. The repo-level cc-by-4.0 tag reflects the curation of the collection, not a uniform document license. Always check the per-document `metadata.rights` field (and the source at `metadata.url`) before redistributing or using individual documents commercially.

Document JSON Schema

json

{
  "metadata": {
    "gardian_id": "string",
    "source": "string",
    "url": "string",
    "description": "string",
    "id": "string",
    "title": "string",
    "language": "string",
    "resource_type": "string",
    "release_year": "string",
    "rights": "string",
    "dataNODE_id": "string",
    "keywords": [ "string" ],
    "geography": {
      "regions": [ "string" ],
      "countries": [ "string" ]
    }
  },
  "keywords": "null",
  "sieverID": "string",
  "pagecount": "string",
  "content": "string",
  "tokenCount": "string",
  "images": [],
  "tables": []
}

License

This dataset is released under CC-BY-4.0.

Acknowledgements

This dataset was developed for the Generative AI for Agriculture (GAIA) project, funded by the Gates Foundation and UK International Development (FCDO), in collaboration between CGIAR and SCiO.

Generated automatically on 2026-07-31 02:55:20 Source directory: ./sieveroutputifpri