Nyries/middle-east-geopolitics-corpus

Middle East Geopolitics Corpus

A cleaned and deduplicated corpus of ~7,800 documents specializing in Middle East geopolitics, designed to power a RAG (Retrieval-Augmented Generation) pipeline.

Description

This dataset contains articles and reports from think tanks, academic institutions, and encyclopedias covering Middle East geopolitics: conflicts, diplomacy, regional actors (Israel, Iran, Saudi Arabia, Turkey, Palestine), and external powers (United States, Russia, China).

Sources

Data Cleaning Pipeline

Steps applied in order:

1. 1.Exact deduplication — duplicate removal via SHA-256 hash
2. 2.Fuzzy deduplication — MinHash LSH (Jaccard ≥ 0.85, 128 permutations)
3. 3.Language filtering — English-only documents retained (langdetect)
4. 4.Repetition filtering — documents with repetition ratio > 0.3 removed
5. 5.Boilerplate removal — recurring headers, footers, and navigation menus stripped
6. 6.PII removal — emails and phone numbers anonymized

Result: 7,796 documents / 65.21 MB (from 7,852 raw documents)

Format

.jsonl files (JSON Lines), one document per line:

{
  "title": "Iran's Regional Strategy",
  "content": "...",
  "url": "https://...",
  "source": "Brookings Institution",
  "date": "2023-04-15"
}

Usage

from datasets import load_dataset

dataset = load_dataset("Nyries/middle-east-geopolitics-corpus")

For the full RAG pipeline, see the GitHub repository.

Chars/Tokens Ratio

The slightly lower ratio compared to FineWeb-Edu is explained by the prevalence of geopolitical proper nouns and acronyms (IRGC, GCC, P5+1) that tokenize into multiple subword units.

Model Trained on This Corpus

Nyries/qwen3-0.8b-middle-east-cpt — Continuous pre-training of Qwen3.5-0.8B-Base on this corpus.

License

Original documents are subject to their respective source licenses. This compiled dataset is distributed under CC BY 4.0.