CoolFace
Datasetpublic

odoma/reference-parsing-finetuning

Reference Parsing Finetuning Dataset A fine-tuning dataset for bibliographic reference extraction and parsing, combining LinkedBooks, CEX, and EXCITE datasets into conversation-style examples for LLM SFT. Dataset Description This dataset teaches models to extract and parse bibliographic references from text into structured JSON format. Examples follow a conversational format with system/user/assistant messages, using various prompt variants for diversity. Data… See the full description on the dataset page: https://huggingface.co/datasets/odoma/reference-parsing-finetuning.

sourceHugging Faceupdated 8mo agoView on Hugging Face
0likes16downloads
Dataset Card

Reference Parsing Finetuning Dataset

A fine-tuning dataset for bibliographic reference extraction and parsing, combining LinkedBooks, CEX, and EXCITE datasets into conversation-style examples for LLM SFT.

Dataset Description

This dataset teaches models to extract and parse bibliographic references from text into structured JSON format. Examples follow a conversational format with system/user/assistant messages, using various prompt variants for diversity.

Data Sources:

  • —LinkedBooks: Multi-language reference strings with structured metadata
  • —CEX: English academic papers with TEI XML parsed references
  • —EXCITE: Multi-language academic papers with parsed references

Data Fields

FieldTypeDescription
messageslistConversation messages with role (system/user/assistant) and content
sourcestringData source: linkedbook, cex, or excite
splitstringDataset split: train or valid
modestringExample type: single (1 reference) or group (multiple references)
languagestringLanguage code (e.g., en, de, fr)
ref_countintNumber of references in the example
file_idstring\nullSource document ID (for CEX/EXCITE)
categorystring\nullDocument category (for CEX)

Splits

SplitExamplesDescription
train~1,708Main training data
valid~115Validation set

Distribution:

  • —~70% single-reference examples, ~30% multi-reference groups
  • —~10-15% LinkedBook, ~30-35% CEX, ~50-55% EXCITE

Data Creation and Processing

  1. 1.Data Loading: Loads references from LinkedBooks (Training and Validation JSONL), CEX (JSON + TEI XML), and EXCITE (JSON + XML)
  2. 2.Validation: Filters invalid references (missing titles/authors, unparsed authors, mismatched counts)
  3. 3.Sampling: Stratified sampling by category/class (30% train rate for CEX/EXCITE)
  4. 4.Grouping: Groups references into batches (3-20 refs per group with weighted probabilities)
  5. 5.Prompt Variants: Applies 5 prompt variants with weighted distribution (40% detailed, 25% minimal, 25% task-based, 5% ultra-minimal, 5% no prompt)
  6. 6.Format Conversion: Converts to conversation-style format with structured JSON output

Credits

The dataset is being developed by Yurui Zhu (Odoma). This work is carried out in the context of the EU-funded GRAPHIA project (grant ID: 101188018).