ritikraj2425/ai-research-papers

AI Research Papers: Metadata & Research Trends (2018–2026)

Dataset Description

AI Research Papers: Metadata & Research Trends is a large-scale collection of research paper metadata focused on Artificial Intelligence and related research areas.

The dataset was collected programmatically from the OpenAlex API and covers research papers published between 2018 and 2026.

It combines bibliographic metadata, abstracts, authors, institutions, countries, research topics, citation information, open-access information, and external identifiers into a single structured dataset.

No manual data entry or manual labeling was used during dataset creation.

The dataset is designed to be useful for:

• —AI research trend analysis
• —Bibliometric analysis
• —Citation analysis
• —Academic search and discovery
• —Research-topic analysis
• —Natural Language Processing (NLP)
• —Exploratory Data Analysis (EDA)
• —Academic recommendation systems
• —Research analytics
• —Studying the evolution of AI research

Dataset Statistics

Data Source

The dataset is derived from OpenAlex, an open catalog of scholarly works, authors, institutions, sources, and related scholarly metadata.

Source:

https://openalex.org/

The data was retrieved programmatically through the OpenAlex API rather than by scraping individual publisher websites.

Features

The dataset contains the following fields.

Paper Identification

• —openalex_id — Unique OpenAlex identifier for the paper.
• —doi — Digital Object Identifier when available.
• —title — Title of the research paper.

Publication Information

• —publication_date — Publication date.
• —publication_year — Publication year.
• —publication_month — Publication month.
• —type — OpenAlex work type.
• —language — Language of the publication.

Abstract

• —abstract — Paper abstract when available.
• —has_abstract — Indicates whether an abstract is available.
• —abstract_word_count — Number of words in the abstract.

Abstracts supplied by OpenAlex may be represented internally as an inverted index. These were reconstructed into readable text during preprocessing.

Authors

• —authors — List of authors associated with the paper.
• —author_count — Number of authors.

Institutions and Countries

• —institutions — Institutions associated with the paper's authors.
• —countries — Countries associated with the authors.

Research Topics

• —primary_topic — Primary research topic assigned by OpenAlex.
• —topics — Research topics associated with the paper.
• —keywords — Available keywords.
• —primary_topic_id — OpenAlex identifier for the primary topic.
• —primary_subfield — Primary research subfield.
• —primary_field — Primary research field.
• —primary_domain — Primary research domain.

These fields make it possible to study the distribution and evolution of different areas of AI and related research.

Publication Venue

• —venue — Journal, conference, repository, or other publication venue.
• —publisher — Publisher associated with the venue.

Citation Information

• —citation_count — Citation count recorded by OpenAlex.
• —referenced_works_count — Number of works referenced by the paper.

Citation counts are dynamic and may change as OpenAlex updates its records.

Open Access

• —is_open_access — Whether the work is marked as open access.
• —oa_status — Open-access status.
• —has_fulltext — Indicates whether full-text information is available.
• —has_pdf — Indicates whether a PDF location is available.

External Identifiers

• —has_doi — Whether a DOI is available.
• —has_pmid — Whether a PubMed identifier is available.
• —has_pmcid — Whether a PubMed Central identifier is available.
• —arxiv_id — arXiv identifier when available.
• —github_url — GitHub repository URL when available.

Derived Features

• —title_word_count — Number of words in the paper title.
• —abstract_word_count — Number of words in the abstract.
• —has_abstract — Boolean indicating abstract availability.
• —publication_month — Month extracted from the publication date.

Research Areas

The collection covers papers associated with AI and related research areas, including but not limited to:

• —Artificial Intelligence
• —Machine Learning
• —Deep Learning
• —Natural Language Processing
• —Computer Vision
• —Large Language Models
• —Transformers
• —Generative AI
• —Diffusion Models
• —Reinforcement Learning
• —Multimodal AI
• —AI Agents
• —Robotics

Because the dataset is based on scholarly metadata and topic classifications, it may also contain papers from interdisciplinary areas where AI or machine learning techniques are applied.

Data Collection

The dataset was generated through an automated data collection pipeline using the OpenAlex API.

The general pipeline was:

OpenAlex API
      ↓
AI/ML research paper retrieval
      ↓
Cursor-based pagination
      ↓
Raw data storage
      ↓
Metadata extraction
      ↓
Abstract reconstruction
      ↓
Data normalization
      ↓
Deduplication
      ↓
Derived feature generation
      ↓
CSV + Parquet

No manual labeling was performed.

Data Processing

The following preprocessing steps were applied:

1. 1.Retrieved paper metadata from OpenAlex.
2. 2.Reconstructed abstracts from OpenAlex abstract data when available.
3. 3.Extracted authors and author counts.
4. 4.Extracted institutions and countries.
5. 5.Extracted research topics and keywords.
6. 6.Extracted publication venue and publisher information.
7. 7.Extracted citation and reference information.
8. 8.Extracted open-access information.
9. 9.Extracted external identifiers such as DOI and arXiv IDs.
10. 10.Normalized missing and inconsistent values.
11. 11.Removed duplicate records using the OpenAlex identifier.
12. 12.Generated additional statistical features such as title and abstract word counts.

File Formats

The dataset is provided in two formats:

CSV

A standard comma-separated format suitable for:

• —Python
• —R
• —Excel
• —SQL imports
• —General data analysis

Parquet

Parquet is provided as an efficient columnar representation suitable for:

• —Python/Pandas
• —PyArrow
• —Hugging Face Datasets
• —Large-scale data processing
• —Analytical workflows

For programmatic use, the Parquet version is recommended.

Example

A record contains information similar to:

Title:
Transforming Science with Large Language Models

Publication Year:
2026

Type:
article

Language:
en

Primary Topic:
Scientific Computing and Data Management

Citation Count:
8

Open Access:
true

Potential Use Cases

1. AI Research Trends

Analyze how research activity has changed between 2018 and 2026.

For example:

• —Number of papers per year
• —Growth of different AI topics
• —Emergence of LLM research
• —Growth of generative AI
• —Changes in research domains

2. Bibliometric Analysis

Study:

• —Citation distributions
• —Highly cited papers
• —Author productivity
• —Institutional contributions
• —Publication venues

3. Academic Search

The title, abstract, keywords, and topic information can be used to build academic search and discovery systems.

4. NLP

The title and abstract fields can be used for:

• —Text classification
• —Topic modeling
• —Clustering
• —Semantic search
• —Information retrieval
• —Research recommendation

5. Research Recommendation

The metadata can be used as a starting point for building systems that recommend papers based on:

• —Topics
• —Keywords
• —Abstract similarity
• —Authors
• —Research fields

Limitations

This dataset should not be considered a complete representation of all AI research.

Its coverage depends on the papers and metadata indexed by OpenAlex and on the collection criteria used during dataset creation.

Some records may contain missing or incomplete:

• —Abstracts
• —Author information
• —Institution information
• —DOI
• —arXiv identifiers
• —GitHub URLs
• —Open-access information

Citation counts are not static and may change when OpenAlex updates its data.

The year 2026 represents only the publications available at the time the dataset was collected and therefore does not represent the complete 2026 publication year.

Topic classifications and other metadata are provided by OpenAlex and should be interpreted as metadata rather than manually verified labels.

License and Attribution

This dataset is derived from OpenAlex scholarly metadata.

Please acknowledge OpenAlex when using this dataset and consult the current OpenAlex documentation and licensing information for the underlying data.

OpenAlex:

https://openalex.org/

Citation

If you use this dataset in a project, analysis, research work, or educational material, please cite this dataset and acknowledge OpenAlex as the underlying data source.

Intended Audience

This dataset is intended for:

• —Students
• —Researchers
• —Data scientists
• —Machine learning practitioners
• —NLP researchers
• —Bibliometric researchers
• —Developers building academic search or recommendation systems
• —Anyone interested in analyzing AI research trends

Version

Version: 1.0

Coverage: 2018–2026

Source: OpenAlex API

Collection: Automated