birdie0111/SciCiteVal
SciCiteVal Introduction Our dataset is specifically designed for citation verification, it contains manual annotations labeling citations as Correct, Incorrect or Unrelated. For citations labeled Incorrect, we further define five detailed sub-categories to characterize the nature of the inaccuracy. Each data sample is structured as a pair, consisting of the citation context from the citing paper and the corresponding evidence passage from the cited paper that… See the full description on the dataset page: https://huggingface.co/datasets/birdie0111/SciCiteVal.
SciCiteVal
Introduction
Our dataset is specifically designed for citation verification, it contains manual annotations labeling citations as `Correct, Incorrect or Unrelated. For citations labeled Incorrect`, we further define five detailed sub-categories to characterize the nature of the inaccuracy. Each data sample is structured as a pair, consisting of the citation context from the citing paper and the corresponding evidence passage from the cited paper that justifies the label.
Our dataset contains 1,034 citations distributed across three categories: 302 Correct citations, 302 Incorrect citations, and 430 Unrelated citations. The citations are from scientific papers in machine learning and biology domains. Both Correct and Incorrect citations are adapted from the QASA dataset, whereas unrelated citations are extracted from real citations that cite unrelated articles.
Language
English
Supported Tasks
- Text Classification
Files Stucture
- Few_shot_matrix contains confusion-matrix figures of each benchmarking model with few-shot prompt.
- Results_cleaned contains benchmarking results of all the models with zero-shot and few-shot prompts in tsv format.
- Zero_shot_matrix contains confusion-matrix figures of each benchmarking model with zero-shot prompt.
- experiment_dataset.tsv is the file that we use as input for benchmarking models
- Qasa_annotated.tsv is the file that contains details of annotations and modifications we made with Qasa dataset. This file preserves the original QASA dataset IDs.
Dataset Information
The dataset contains four columns: "Citation Context", "Cited Content", "Label", and "Distortion Category"
- Citation_context (String) contains the citation context transformed from the "Composition" of the Qasa dataset by our annotators
- Cited_content (String) contains the content that justifies the corresponding citation context, originally from the "Evidence" of the Qasa dataset
- Label (String) contains the label for the corresponding citation.
- Twist_category (String) contains the label of distortion for the manually-distorted citation context.
Annotation Process
Correct Citations: We manually verified and transformed QASA "composition" to citation. Compositions consisting only of "Yes"/"No" answers were excluded.
Incorrect Citations: Two annotators with NLP backgrounds read the original Qasa "composition" and systematically distorted it to produce a citation context that inaccurately references the "Cited Content". Minimal changes were made to improve fluency.
Unrelated Citations: These were manually collected from scientific papers across diverse fields (e.g., social science, biology). Only complete sentences from the main body of the text were included, content from tables or figures was excluded.
Referenced Dataset
Citation Information
Will be updated soon