ines-besrour/unarxive_2024

Example Preview

Here is a small excerpt of the dataset format: 📄 preview.jsonl

Dataset Card for UnarXive 2024

UnarXive 2024 is a large-scale, structured dataset of 2.3 million full-text arXiv papers (1991–2024), processed for use in NLP and information retrieval tasks. Each paper is provided in a structured JSONL format.

Features

• —2.28 million structured papers across physics, CS, mathematics, and other fields
• —Logical section structure (Introduction, Methods, etc.) for discourse-aware modeling
• —Metadata-rich: title, authors, abstract, arXiv ID, categories, license, language, citedbycount
• —In-text citation markers and linked reference entries
• —30M+ citation links with 51.2% resolved to OpenAlex IDs
• —Preserved mathematical expressions, figures, and tables with positional spans
• —Multilingual content, mostly in English, but includes French, German, Catalan, and more (25+ languages)

Usage

The dataset supports tasks such as citation recommendation, scientific QA, retrieval-augmented generation (RAG), Citation function and polarity classification and paper embedding training.

Pipeline

GitHub: ines-besrour/UnarXive-2024

Dataset licensed under the MIT License.