ethicalabs/Research-Intent-Judge

Research Intent — LLM-as-Judge

    


**▶️ Watch the Video**

LLM-as-Judge annotations for research paper intent classification, collected through the Echo-DSRN collaborative platform during the OpenAIRE AI Hackathon 2026.

The dataset has one split per judge model (Gemma_4_E4B_it_GGUF, Qwen3.6_35B_A3B_GGUF, Bonsai_8B_gguf, ...) plus a human_annotations split with curator annotations. Split names use underscores in place of the dashes in model names (HF does not allow dashes in split names).

Each record is a single judge verdict from one model on one paper, in ChatML conversation format with the judge's reasoning (rationale) included both inline in the assistant message and as a separate reasoning field.

OpenAIRE AI Hackathon 2026 🇪🇺

This dataset is part of ethicalabs.ai's entry in the OpenAIRE AI Hackathon 2026, co-organised by OpenAIRE and Alien Intelligence — a 12-week open science build challenge.

[🔴 Live Demo → openaire-2026.ethicalabs.ai](https://openaire-2026.ethicalabs.ai/)

The live application classifies research papers in real time using this model, with publication metadata streamed from the OpenAIRE Graph API under CC BY 4.0.

Labels

• —Methodology — introduces a new algorithm, model architecture, or method
• —Dataset — primarily contributes a new dataset, benchmark, or annotation scheme
• —Review — survey, systematic review, or position paper
• —Applied — applies existing methods to a domain problem
• —Theoretical — mathematical or formal analysis
• —Unclassifiable — garbled, non-English, or non-academic content

Data structure

{
  "doi": "10.48550/arXiv.2106.01345",
  "title": "...",
  "description": "abstract text",
  "initial_intent": "Methodology",
  "messages": [
    {"role": "system", "content": "You are a multilingual research paper intent classifier..."},
    {"role": "user", "content": "Classify the research intent of this paper:\n\nTitle: ...\nAbstract: ..."},
    {"role": "assistant", "content": "The paper introduces a novel...\n\nDataset"}
  ],
  "reasoning": "The paper introduces a novel...",
  "model_prediction": "Dataset",
  "is_flagged": false,
  "flag_reason": null,
  "confidence": "high"
}

• —messages — ChatML conversation with inline reasoning in the assistant turn
• —reasoning — the judge's rationale as a standalone field
• —model_prediction — the label proposed by the judge
• —is_flagged / flag_reason — quality-issue signal (garbled text, language mismatch)
• —confidence — high / medium / low

Splits

One split per judge model (21 models, ~10k records each) plus human_annotations (46 curator annotations). Records within a split are per-paper single verdicts; a paper appears in every model split that judged it.

Provenance

Papers were collected from arXiv and OpenAIRE via the OpenAIRE Graph and judged by 20+ open-weight LLMs. Underperforming models (e.g. DeepSeek-Qwen3-8B, Kurtis-E1.1, Ministral-3-3B, Jan-v1-4B) are excluded. MTP and non-MTP variants of the same model are merged (they produce identical predictions).

Export tooling: scripts/export_llm_judge_dataset.py in the OpenAIRE-AI-Research-Evaluator repository.

License

CC-BY 4.0. Paper metadata originates from OpenAIRE Graph (CC BY 4.0).

Citation

If you reuse this work, please cite:

@software{echo_dsrn_research_intent_clf,
  author = {Massimo Roberto Scamarcia},
  title = {OpenAIRE Graph Evaluator: Research paper classifier and LLM-as-Judge pipeline},
  year = {2026},
  url = {https://huggingface.co/collections/ethicalabs/openaire-ai-hackathon-2026}
}