IT-ZBMED/Agriculture_NER_Dataset_for_FAIR_Metadata_Enrichment

Dataset Card for A Manually Annotated Agricultural Dataset for AI-Based NER and FAIR Metadata Enrichment

<!-- Provide a quick summary of the dataset. -->

Supported by FAIRagro, the pilot use case “Increasing FAIRness of FAIRagro data through AI-supported metadata enrichment” addresses this gap by creating a manually annotated text corpus designed to support Named Entity Recognition (NER) models in agricultural research. NER models can automate metadata extraction from unstructured text, such as dataset abstracts, thereby enabling metadata enrichment.

Dataset Details

Dataset Description

<!-- Provide a longer summary of what this dataset is. -->

This dataset contains Named Entity Recognition (NER) annotations derived from curated CAS XMI files exported from INCEpTION. The corpus is provided in three complementary formats, each optimized for different model architectures and evaluation scenarios:

1. 1.Document-level tokenized (file-based) CSV
2. 2.Sentence-level tokenized CSV

Annotation Entities, Their Attributes, and Definitions

Labels Mapping

Label2id

{
  "O": 0,
  "B-soilReferenceGroup": 1,
  "I-soilReferenceGroup": 2,
  "B-soilOrganicCarbon": 3,
  "I-soilOrganicCarbon": 4,
  "B-soilTexture": 5,
  "I-soilTexture": 6,
  "B-startTime": 7,
  "I-startTime": 8,
  "B-endTime": 9,
  "I-endTime": 10,
  "B-city": 11,
  "I-city": 12,
  "B-duration": 13,
  "I-duration": 14,
  "B-cropSpecies": 15,
  "I-cropSpecies": 16,
  "B-soilAvailableNitrogen": 17,
  "I-soilAvailableNitrogen": 18,
  "B-soilDepth": 19,
  "I-soilDepth": 20,
  "B-region": 21,
  "I-region": 22,
  "B-country": 23,
  "I-country": 24,
  "B-longitude": 25,
  "I-longitude": 26,
  "B-latitude": 27,
  "I-latitude": 28,
  "B-cropVariety": 29,
  "I-cropVariety": 30,
  "B-soilPH": 31,
  "I-soilPH": 32,
  "B-soilBulkDensity": 33,
  "I-soilBulkDensity": 34
}

id2label

{
  "0": "O",
  "1": "B-soilReferenceGroup",
  "2": "I-soilReferenceGroup",
  "3": "B-soilOrganicCarbon",
  "4": "I-soilOrganicCarbon",
  "5": "B-soilTexture",
  "6": "I-soilTexture",
  "7": "B-startTime",
  "8": "I-startTime",
  "9": "B-endTime",
  "10": "I-endTime",
  "11": "B-city",
  "12": "I-city",
  "13": "B-duration",
  "14": "I-duration",
  "15": "B-cropSpecies",
  "16": "I-cropSpecies",
  "17": "B-soilAvailableNitrogen",
  "18": "I-soilAvailableNitrogen",
  "19": "B-soilDepth",
  "20": "I-soilDepth",
  "21": "B-region",
  "22": "I-region",
  "23": "B-country",
  "24": "I-country",
  "25": "B-longitude",
  "26": "I-longitude",
  "27": "B-latitude",
  "28": "I-latitude",
  "29": "B-cropVariety",
  "30": "I-cropVariety",
  "31": "B-soilPH",
  "32": "I-soilPH",
  "33": "B-soilBulkDensity",
  "34": "I-soilBulkDensity"
}

Dataset Versions

1️⃣ Document-Level Tokenized Format (File-Based)

Each row in this CSV corresponds to a complete document, tokenized using spaCy.

Columns

Intended Use

• —Document-level transformer models (Longformer, BigBird, etc.)
• —Corpus statistics and label distribution analysis
• —Document classification + NER pipelines

2️⃣ Sentence-Level Tokenized Format

Each row corresponds to a single sentence, preserving alignment with the original document.

Sentence Identifier Format

fileID-sentenceIndex
Example:
73465-03

Columns

Same as file-level format, with:

Intended Use

• —Classical BERT-style NER (max length ≈512 tokens)
• —Models with fixed-length input windows
• —Fine-grained sentence-level training and evaluation

Code Repository

All scripts used to preprocess the INCEpTION XMI files, generate the tokenized datasets, convert annotations into BIO format, create JSON span annotations, and build the HuggingFace-ready dataset version are openly available in the following GitHub repository:

🔗 Dataset Processing Code Repository: https://github.com/fairagro/pilot-uc-textmining-metadata The current version is v1.0.0

The repository includes:

• —Fetching the metadata from the research data infrastructures
• —CAS → BIO conversion scripts
• —Sentence and document tokenization routines
• —Gazetteer-based location normalization
• —DOI mapping utilities
• —Span annotation builder (HuggingFace-like JSON format)
• —Dataset export pipeline (CSV, JSON, HuggingFace)
• —Example configuration files
• —A full reproducible workflow for regenerating the dataset

Authors and Affiliations

Licensing

The FAIRagro Metadata Enrichment NER Dataset is released under the:

Creative Commons Attribution 4.0 International (CC BY 4.0) License

License URL:

https://creativecommons.org/licenses/by/4.0/

Cite as:

@dataset{abdelmalak_fairagro_ner_2025,
  author       = {Abdelmalak, Abanoub and Schneider, Gabriel and Riegler, Heike and Meier, Kristin and Specka, Xenia and Svoboda, Nikolai and Husain, Murtuza and Fluck, Juliane},
  title        = {{FAIRagro NER Dataset: Increasing FAIRness of FAIRagro Data Through AI-Supported Metadata Enrichment}},
  year         = {2025},
  publisher    = {Fachrepositorium Lebenswissenschaften (FRL)},
  doi          = {10.4126/FRL01-6526458},
  url          = {https://doi.org/10.4126/FRL01-6526458},
  note         = {Version 1.0}
}