datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
faiss-smollm
FAISS-Based Novelty Detection for SmolLM and SmolLM2
This tutorial demonstrates how to the measure novelty of text queries with respect to the provided SmolLM and SmolLM2 pretraining corpora, with optional ColBERTv2 re-ranking for improved precision.
Overview
The pipeline consists of four main steps:
Generate Embeddings - Encode your queries using a sentence transformer
FAISS Search - Retrieve top-K most similar documents from the pretraining corpus
Combine… See the full description on the dataset page: https://huggingface.co/datasets/stai-tuebingen/faiss-smollm.pubmed-faiss-indexesfaiss_indexThe faiss index is built for fast retrieval using the embedding computed by ProTrek_650M_UniRef50.
Please download the faiss index and put it into the faiss_index directory, i.e. faiss_index/SwissProt/.... For more details, see our repo.
faiss_indicesfaiss-integration-testpixelrag-faiss-indexespubchem-faiss-library
PubChem-Scale Molecule Retrieval Library (FAISS)
This dataset provides a prebuilt, global FAISS retrieval library for molecule search at PubChem scale.
It includes molecular metadata (SMILES, formula, exact mass), precomputed embeddings, and FAISS indices.
Contents
index_smi.faiss: FAISS HNSW index built on SMI-TED embeddings (vectors_smi.npy).
index_chem.faiss: FAISS HNSW index built on ChemBERTa embeddings (vectors_chem.npy).
vectors_smi.npy: SMI-TED embeddings for… See the full description on the dataset page: https://huggingface.co/datasets/YinkaiW/pubchem-faiss-library.wikipedia_en_top_nopic_faissbio-faiss-longevity-v1
bio-faiss-longevity-v1
A FAISS index + metadata for scientific retrieval
Contents
index.faiss: FAISS index (cosine w/ inner product).
meta.jsonl: one JSON per chunk; fields include chunk_id, paper_id, title, section, subsection, paragraph_index, keywords, boost.
index.info.json: (optional) dimensions, index type, faiss version.
Build provenance
Chunking: hierarchical (section→paragraph→~480-token chunks, ~15% overlap)
Embedder:… See the full description on the dataset page: https://huggingface.co/datasets/bio-protocol/bio-faiss-longevity-v1.pubmed-faiss-indexneophyte-faiss-index-v1
neophyte-faiss-index-v1
A FAISS index + metadata for scientific retrieval
Contents
index.faiss: FAISS index (cosine w/ inner product).
meta.jsonl: one JSON per chunk; fields include chunk_id, paper_id, title, section, subsection, paragraph_index, keywords, boost.
index.info.json: (optional) dimensions, index type, faiss version.
Build provenance
Chunking: hierarchical (section→paragraph→~480-token chunks, ~15% overlap)
Embedder:… See the full description on the dataset page: https://huggingface.co/datasets/bio-protocol/neophyte-faiss-index-v1.faiss_indexingabstracts-faiss
abstracts-faiss
This is a faiss index, trained on abstracts-embeddings. A ready-to-go search interface for using this index is available at abstracts-index.
Building
It was trained with the train.py script found at abstracts-search with the options -N -c 65536 (normalized, train 65536 clusters), using the default preprocess technique OPQ96_384 (PCA to a 384-dimensional vector, then apply OPQ for a 96-byte code). Note that, although the Stella model was trained with… See the full description on the dataset page: https://huggingface.co/datasets/colonelwatch/abstracts-faiss.eduintel-faiss
EduIntel FAISS Index v1
A FAISS vector index of Sepedi (Sesotho sa Leboa) text chunks for the
EduIntel RAG pipeline, built by Sediba AI.
This is not a model — it is a retrieval index. It stores chunked Sepedi text and their
embeddings so that a retriever can find relevant Sepedi content for a query, which is then
fed to a generative model (e.g. Sedibaai/SedibaLM)
for answer generation.
Contents
File
Purpose
eduintel.index
The FAISS index (flat or IVF… See the full description on the dataset page: https://huggingface.co/datasets/Sediba-AI/eduintel-faiss.EarthLoc2_FAISS
🌍 EarthLoc2 FAISS Index (DINOv2 + SALAD)
This is the FAISS index used in the Hugging Face Space:🔗 EarthLoc2: Image Geolocalization
📦 What’s Inside?
This index was constructed by encoding all of the 2021 Sentinel-2 satellite images from the EarthLoc dataset:
👉 EarthLoc_2021_Database
Each image is encoded four times (with rotations of 0°, 90°, 180°, and 270°), and descriptors were extracted using:
https://huggingface.co/pawlo2013/EarthLoc2
Also see the corresponding… See the full description on the dataset page: https://huggingface.co/datasets/pawlo2013/EarthLoc2_FAISS.plant-disease-faiss-indexbio-faiss-d1ckgpt-v1
bio-faiss-d1ckgpt-v1
A FAISS index + metadata for scientific retrieval
Contents
index.faiss: FAISS index (cosine w/ inner product).
meta.jsonl: one JSON per chunk; fields include chunk_id, paper_id, title, section, subsection, paragraph_index, keywords, boost.
Build provenance
Chunking: hierarchical (section→paragraph→~380-token chunks, ~15% overlap)
Embedder: bio-protocol/scientific-retriever (mean-pooled, L2-normalized)
Similarity: cosine via inner… See the full description on the dataset page: https://huggingface.co/datasets/bio-protocol/bio-faiss-d1ckgpt-v1.msmarco-passage-train__tct_colbert__faiss2opq__M96_nbits8__ps159744__neg200__ibn__lr__
msmarco-passage-train__tct_colbert__faiss2opq__M96_nbits8__ps159744__neg200__ibn__lr__
Description
This is the PyTerrier JPQIndex for MSMARCO v1 passage corpus, which corresponds to an result from the SIGIR 2026 reproducibility paper.
Usage
# Load the artifact
import pyterrier as pt
import pyterrier_dr
index = pt.Artifact.from_hf('jpq-repro/msmarco-passage-train__tct_colbert__faiss2opq__M96_nbits8__ps159744__neg200__ibn__lr__')
model =… See the full description on the dataset page: https://huggingface.co/datasets/jpq-repro/msmarco-passage-train__tct_colbert__faiss2opq__M96_nbits8__ps159744__neg200__ibn__lr__.nq-train__tct_colbert__faiss2opq__M96_nbits8__ps159744__neg200__ibn__lr__
nq-train__tct_colbert__faiss2opq__M96_nbits8__ps159744__neg200__ibn__lr__
Description
This is the PyTerrier JPQIndex for the Wikipedia 2018 corpus used by Natural Questions (NQ).
Usage
# Load the artifact
import pyterrier as pt
import pyterrier_dr
index = pt.Artifact.from_hf('jpq-repro/nq-train__tct_colbert__faiss2opq__M96_nbits8__ps159744__neg200__ibn__lr__')
model = pyterrier_dr.TctColBert.hnp()
model.model =… See the full description on the dataset page: https://huggingface.co/datasets/jpq-repro/nq-train__tct_colbert__faiss2opq__M96_nbits8__ps159744__neg200__ibn__lr__.charity-faiss-indexwiki-faissWiki_Faiss_Indexes
dataset_info:
features:
- name: text
dtype: string
- name: embeddings
dtype: float32
shape: [384]
configs:
- config_name: default
data_files: "*.parquet"
Wikipedia IVF-OPQ-PQ Vector Database (GPU-Optimized)
A high-performance, GPU-accelerated FAISS vector database built from Wikipedia articles with pre-computed embeddings. This dataset contains approximately 35 million Wikipedia articles with 384-dimensional embeddings using the all-MiniLM-L6-v2… See the full description on the dataset page: https://huggingface.co/datasets/Ram-G/Wiki_Faiss_Indexes.bio-faiss-microbiome-v1
bio-faiss-microbiome-v1
A FAISS index + metadata for scientific retrieval
Contents
index.faiss: FAISS index (cosine w/ inner product).
meta.jsonl: one JSON per chunk; fields include chunk_id, paper_id, title, section, subsection, paragraph_index, keywords, boost.
Build provenance
Chunking: hierarchical (section→paragraph→~380-token chunks, ~15% overlap)
Embedder: bio-protocol/scientific-retriever (mean-pooled, L2-normalized)
Similarity: cosine via inner… See the full description on the dataset page: https://huggingface.co/datasets/bio-protocol/bio-faiss-microbiome-v1.arxiv_abstracts_specter_faiss_flat_indexversion https://git-lfs.github.com/spec/v1
oid sha256:98b45ea81164d1e1a1dd82255207053b15cd6c69d922a1c5cf3387ce604d4b74
size 28
wikipedia-2024-06-bge-m3-faiss-ivfCaselaw_Access_Project_FAISS_index
The Caselaw Access Project
In collaboration with Ravel Law, Harvard Law Library digitized over 40 million U.S. court decisions consisting of 6.7 million cases from the last 360 years into a dataset that is widely accessible to use. Access a bulk download of the data through the Caselaw Access Project API (CAPAPI): https://case.law/caselaw/
Find more information about accessing state and federal written court decisions of common law through the bulk data service documentation here:… See the full description on the dataset page: https://huggingface.co/datasets/free-law/Caselaw_Access_Project_FAISS_index.quantized_wiki_faissFAISS-v2.0
Dataset Card for NextGLab FAISS-v2.0
The NextGLab FAISS-v2.0 dataset is a dense vector index designed for Retrieval-Augmented Generation (RAG) tasks. It is constructed by embedding text from:
srsRAN Project: A complete 5G RAN solution featuring an ORAN-native CU/DU.
O-RAN Alliance specifications
Summary
Total Documents: 88,808
Total Words: 7,236,372
Embedding Model: BAAI/bge-large-en-v1.5
Vector Store: FAISS (Facebook AI Similarity Search)
Developed by: NextG lab@ NC… See the full description on the dataset page: https://huggingface.co/datasets/NextGLab/FAISS-v2.0.govgis_nov2023-slim-faiss-index
govgis_nov2023-slim-faiss-index
The native, LangChain-free FAISS retrieval artifact for
joshuasundance/govgis_nov2023-slim-faiss,
produced by that Space's Stage 2 modernization (real one-time conversion of
the legacy pickle-like artifact, run in an isolated, network-disabled
process — see
docs/modernization-plan.md
and docs/stage0/
in that repo for the full methodology and evidence).
Files
index.faiss — native FAISS IndexFlatIP, 865,304 × 1024-dim vectors… See the full description on the dataset page: https://huggingface.co/datasets/joshuasundance/govgis_nov2023-slim-faiss-index.vietnamese-legal-faiss