CoolFace
Datasetpublic

odoma/gotriple-pretraining-dataset

GoTriple Pretraining dataset Summary The GoTriple Pre-training Dataset is a multilingual corpus built from open-access research artefacts harvested via the GoTriple platform. It focuses on Social Sciences and Humanities (SSH) content, addressing their limited presence in standard LLM pre-training corpora. Current release includes History, Sociology, Environmental Sciences, Psychology and Geography texts (~23.14B tokens). Intended Use Continuous… See the full description on the dataset page: https://huggingface.co/datasets/odoma/gotriple-pretraining-dataset.

sourceHugging Faceccupdated 8mo agoView on Hugging Face
3likes70downloads
Dataset Card

GoTriple Pretraining dataset

Summary

The GoTriple Pre-training Dataset is a multilingual corpus built from open-access research artefacts harvested via the GoTriple platform. It focuses on Social Sciences and Humanities (SSH) content, addressing their limited presence in standard LLM pre-training corpora.

Current release includes History, Sociology, Environmental Sciences, Psychology and Geography texts (\~23.14B tokens).

Intended Use

  • —Continuous pre-training of LLMs for SSH knowledge.
  • —Domain adaptation for reasoning, summarisation, QA, and text understanding tasks.
  • —Multilingual and long-form academic text experimentation.

Not suited for:

  • —High-stakes factual inference without validation.
  • —Model training requiring strict licensing confirmation.

Limitations

  • —PDF extraction inconsistencies.
  • —Multilingual documents (avg. 1.3--1.5 languages per text).
  • —Possible near-duplicates.
  • —Noise from URLs, formulas, encoding issues.

Ethical Considerations

  • —Dataset filtered for open-access, but users must verify licenses.
  • —SSH texts may encode cultural or disciplinary biases.
  • —Language imbalance (European languages dominate).
  • —Care advised for historically/socially sensitive applications.

Data Composition

  • —Source: GoTriple API (19.9M artefacts across 27 disciplines).
  • —Current subset: History, Sociology, Environmental Sciences, Psychology and Geography.
  • —Languages: 120+ detected; top: EN, DE, PL, FR, ES.
  • —Token counts: History 4.47B, Sociology 5.33B, Environmental Sciences 5.21B, Psychology 3.65B, Geography 4.46B (total \~23.14B).

Features: raw text, language IDs, spell-check stats, perplexity, PDF metadata, dedupe cluster IDs.

Collection & Processing

  1. 1.Metadata filtering (OA + PDF availability)
  2. 2.Parallel PDF retrieval (retry + backoff; skip \>500MB)
  3. 3.PyMuPDF text extraction + heuristics
  4. 4.Semantic deduplication (Model2Vec + FAISS, 0.99 similarity)
  5. 5.Quality scoring (GlotLID, spell-check, perplexity)

Version History

  • —v1.0 --- 5 disciplines: History, Sociology, Environmental Sciences, Psychology and Geography (23.14B tokens)
  • —v0.1 --- History + Sociology (\~9.8B tokens)

Credits

This dataset was created by Harshdeep Singh (Odoma) and Matteo Romanello in the context of the EU-funded GRAPHIA project (grant ID: 101188018).