GSMA/Telco-Common-Corpus
Telco Common Corpus (TCC) is a ten billion tokens collection of fully open, free licensed telecommunications knowledge (scientific literature, patents, open data, and open-web projects) with licence and provenance verified at a document-level. TCC stems from GSMA's effort to make AI work for the telecom sector. The Open-Telco LLM Benchmarks and the broader Open Telco AI initiative have already established that current models fall short on real telecom tasks, including network management and… See the full description on the dataset page: https://huggingface.co/datasets/GSMA/Telco-Common-Corpus.
Telco Common Corpus (TCC) is a ten billion tokens collection of fully open, free licensed telecommunications knowledge (scientific literature, patents, open data, and open-web projects) with licence and provenance verified at a document-level.
TCC stems from GSMA's effort to make AI work for the telecom sector. The Open-Telco LLM Benchmarks and the broader Open Telco AI initiative have already established that current models fall short on real telecom tasks, including network management and, even more importantly, that they have not progressed significantly over the last two years.
Composition
What equally sets TCC apart is the diversity of its sources: alongside peer-reviewed articles, it reaches technical reports, standards-adjacent project deliverables, patents, and public-domain government research on propagation, spectrum, and coding.
Processing
Most TCC sources come from doc or PDF documents that require advanced processing. We used a Pleias internal pipeline (Stratum) relying on open weights VLMs like dots.ocr to properly parse each document at the section level and preserve fundamental document structure like tables.
The dataset is almost exclusively English. The few multilingual documents have been detected using <a href="https://huggingface.co/PleIAs/CommonLingua">CommonLingua</a>, a SOTA language detection already jointly released by Pleias and GSMA.