LiamDuero/telco-retrieve-chunks

telcolens-chunks

Raw ingestion artifacts (ChromaDB vector stores + SQLite FTS5 full-text indexes) produced by telcolens (open-telco-rag) while grid-searching RAG configurations over 3GPP Release 19 specifications, for an MSc thesis benchmarking retrieval-augmented generation over telecom standards (GSMA Open Telco AI Initiative). This is not a datasets.load_dataset()- compatible tabular dataset — it's a folder of persisted vector-database directories, one per (embedding model × chunking strategy × enrichment) combination, meant to be pulled individually and opened with chromadb (see Loading a collection below).

[!IMPORTANT] `final-grid/` is the only folder that matters. It's the complete, verified 120/120 run — every embedding model × chunking strategy × enrichment combination, with spec_version metadata patched to be correct across every entry. ingestion-grid/ and final-ingestion-grid/ are earlier, superseded iterations of the same grid, kept only for the thesis's own provenance/reproducibility trail — don't build on them. Every real evaluation result in this project (the 120-combo eval grid, the targeted reranker comparison, the no-RAG baseline) was scored against final-grid/.

Repository structure

The repo totals roughly 2.7 TB across all three grid folders combined — do not snapshot_download() the whole repo. Pull one final-grid/{run_name}/ folder at a time (see below); each is a few GB to a few tens of GB depending on chunking strategy and embedding dimensionality.

The 120 final-grid/ combinations

Every combination is 3GPP Release 19 ("3gpp-r19") text, split along three independent axes:

Embedding models (6):

Chunking strategies (5):

Enrichments (4): none (raw text), metadata_tagging (SDO/working-group/header metadata appended), acronym_expansion (telecom acronym dictionary expansion), llm_metadata (LLM-generated inline entity translations).

6 models × 5 strategies × 4 enrichments = 120 run directories, each named:

3gpp-r19_{model}_{strategy}_{enrichment}_c{chunk_size}_o{overlap}      # or _t{theta} for lumber_chunker

e.g. 3gpp-r19_otel-109m_text_baseline_none_c1024_o200.

Each run directory contains a raw ChromaDB persistence dir (chroma.sqlite3, per-collection HNSW index subfolders) plus a parallel SQLite FTS5 BM25 index ({run_name}_fts.db) and an _ingest_complete.json completion marker — exactly what open-telco-rag ingest writes locally, uploaded as-is.

Loading a collection

This is not loadable with datasets.load_dataset(). Pull one run's folder with huggingface_hub.snapshot_download() (scoped with allow_patterns so you don't fetch the whole 946 GB final-grid/ folder), then open it directly with chromadb:

from huggingface_hub import snapshot_download
import chromadb

run_name = "3gpp-r19_otel-109m_text_baseline_none_c1024_o200"

snapshot_download(
    repo_id="LiamDuero/telcolens-chunks",
    repo_type="dataset",
    allow_patterns=f"final-grid/{run_name}/*",
    local_dir="./pulled_grid",
)

client = chromadb.PersistentClient(path=f"./pulled_grid/final-grid/{run_name}")
collection = client.get_collection(run_name)
print(collection.count(), "chunks")

sample = collection.get(limit=1, include=["documents", "metadatas"])

(telcolens itself does exactly this via open_telco_rag.stores.hf.pull_collection_from_hf() and open_telco_rag.stores.chroma.ChromaStore — see that project's README for the full retrieval/eval pipeline built on top.)

Sample chunk

Pulled live from final-grid/3gpp-r19_otel-109m_text_baseline_none_c1024_o200 (344,285 chunks in this one collection alone — counts vary by strategy/model across the other 119):

{
  "id": "21_series/21201/raw.md::otel-109m::3",
  "metadata": {
    "source_type": "3gpp",
    "series": "21",
    "spec_number": "21.201",
    "canonical_id": "21.201 (Rel-19)",
    "release": "Rel-19",
    "spec_version": "V19.0.0",
    "filename": "21201.md",
    "chunk_index": 3,
    "heading": ""
  }
}

The high-level architecture of such a system is defined in 3GPP TS 23.002 [2] (figure 1b).

## --- 2 References

The following documents contain provisions which, through reference in this text, constitute
provisions of the present document.

- References are either specific (identified by date of publication, edition number, version
  number, etc.) or non-specific.
- For a specific reference, subsequent revisions do not apply.
- For a non-specific reference, the latest version applies. In the case of a reference to a 3GPP
  document (including a GSM document), a non-specific reference implicitly refers to the latest
  version of that document *in the same Release as the present document*.

[1] 3GPP TR 21.905: "Vocabulary for 3GPP Specifications".
[2] 3GPP TS 23.002: "Network architecture".
[3] 3GPP TS 21.900: "Technical Specification Group working methods".

## --- 3 Definitions, symbols and abbreviations

### 3.1 Definitions

Source data & curation

Source text is 3GPP Release 19 technical specifications (Markdown-converted, series 21–38), ingested via telcolens' open-telco-rag ingest pipeline and its master_grid.py grid orchestrator, which parallelizes ingestion across the full (model × strategy × enrichment) cross-product and pushes each completed run directly to this repo. final-grid/ additionally had spec_version metadata corrected post-hoc across all 120 runs (see patch_3gpp_spec_version.py in the telcolens repo) — the reason it supersedes final-ingestion-grid/.

Considerations & limitations

• —Not a standalone corpus. These are pipeline artifacts for a specific benchmarking project, not a general-purpose 3GPP text dataset. If you want the raw 3GPP text itself, go to 3GPP directly.
• —3GPP copyright. The underlying specification text carries 3GPP's own copyright notice ("No part may be reproduced except as authorized by written permission... This Specification is provided for future development work within 3GPP only... shall not be implemented."). Chunks here are retained for RAG research/evaluation purposes, not redistribution as a spec substitute.
• —Superseded folders. ingestion-grid/ and final-ingestion-grid/ may contain incomplete, pre-patch, or otherwise inconsistent runs — they exist for provenance, not reuse.
• —Size. ~2.7 TB total across all three folders; always scope pulls with allow_patterns to a single final-grid/{run_name}/.
• —No PII. Content is exclusively public telecom standards text and pipeline-generated metadata.