GenAI4ELab/papercli-papers

AI Conference & Journal Papers

Searchable metadata and full-text PDF mirrors for papers from top-tier AI venues (NeurIPS, ICML, ICLR, CVPR, ICCV, ECCV, WACV, ACL, EMNLP, NAACL, IJCAI, AAAI, JMLR, Interspeech) from 2023.

• —📊 papers.parquet: The complete dataset containing all fields and all venues.
• —🔍 Per-venue browse views: Easily explore specific subsets by selecting a venue in Subset and a year in Split.

🏗️ Dataset Structure & Storage Strategy

To avoid reaching repository size limits and ensure optimal performance, the project is decoupled into two components:

1. 1.Main Registry (This Repository): Contains the complete searchable metadata, individual index files (papers.parquet), and partitioned per-venue Parquet files (browse/).
2. 2.PDF Storage Shards: The raw PDF binary files are sharded into separate, venue-specific repositories (GenAI4ELab/papercli-papers-[venue]).

🛠️ How to Download PDFs

Because the metadata and actual file pointers reside in this main repository, the standard workflow is to query/filter the metadata here first, then programmatically fetch the corresponding PDF binary from its respective shard.

Python Example

Ensure you have the Hugging Face Hub CLI client installed:

pip install huggingface_hub

You can use the following script to look up a paper and pull its mirrored PDF automatically:

from huggingface_hub import hf_hub_download

# Assuming `row` is a dictionary or pandas row obtained from the metadata Parquet
venue_name = row['venue'].lower()
repo_id = f"GenAI4ELab/papercli-papers-{venue_name}"

path = hf_hub_download(
    repo_id=repo_id,
    filename=row["hf_pdf_path"],
    repo_type="dataset",
)
print(f"Downloaded PDF to: {path}")

🔗 Dataset Hub & Venue Directory

Here is the complete navigation map for the main dataset metadata registry and all corresponding sharded PDF storage repositories:

🛠️ Credits & Tools

This dataset was compiled and structured using [papercli](https://github.com/Keithsel/papercli), an open-source tool designed to index, mirror, and shard academic papers from top-tier AI venues efficiently.

If you find this mirror useful, please consider starring the parent repository and the original papercli project!