singletongue/wikipedia-paragraphs

{
  "id": "en-1980219-1351522064",
  "lang": "en",
  "page_id": 1980219,
  "revision_id": 1351522064,
  "page_type": "article",
  "title": "Éclair",
  "wikidata_qid": "Q273426",
  "paragraph_texts": [
    "An éclair (English: /ɪˈklɛər/ ⓘ ih-KLAIR or /eɪˈklɛər/ ay-KLAIR, French: [eklɛːʁ] ⓘ; lit.\u2009'lightning') is a pastry made with choux dough filled with a cream and topped with a flavored icing. The dough, which is the same as that used for profiteroles, is typically piped into an oblong shape with a pastry bag and baked until it is crisp and hollow inside. Once cool, the pastry is filled with custard (crème pâtissière), whipped cream or chiboust cream, then iced with fondant icing. Other fillings include pistachio- and rum-flavored custard, fruit-flavored fillings or chestnut purée. When the icing is caramel, the dessert may be called a bâton de Jacob (lit.\u2009'Jacob's staff'). A similar pastry in a round rather than oblong shape is called a religieuse.",
    "The word comes from the French éclair, meaning 'flash of lightning', so named because it is eaten quickly (in a flash); however some believe that the name is due to the glistening of the frosting resembling lightning.",
    "The éclair originated during the 19th century in Lyon, France where it was called pain à la Duchesse ('Duchess-style bread') or petite duchesse ('little duchess') until 1850. The word is first attested both in English and in French in the 1860s.",
    ...
  ],
  "paragraph_sections": [
    "__LEAD__",
    "Etymology",
    "History",
    ...
  ],
  "paragraph_wikilinks": [                                                                                                                             [229/346]
    [
      {
        "lang": "en",
        "title": "Pastry",
        "text": "pastry",
        "start": 108,
        "end": 114
      },
      {
        "lang": "en",
        "title": "Choux pastry",
        "text": "choux",
        "start": 125,
        "end": 130
      },
      {
        "lang": "en",
        "title": "Profiterole",
        "text": "profiteroles",
        "start": 237,
        "end": 249
      },
      ...
    ],
    [
      {
        "lang": "en",
        "title": "Lightning",
        "text": "lightning",
        "start": 57,
        "end": 66
      }
    ],
    [
      {
        "lang": "en",
        "title": "Lyon",
        "text": "Lyon",
        "start": 49,
        "end": 53
      }
    ],
    ...
  ],
  "paragraph_html_tags": [
    "p",
    "p",
    "p",
    ...
  ],
  "linked_articles": [
    "France",
    "French cuisine",
    "Pastry",
    ...
  ],
  "categories": [
    "Albanian cuisine",
    "Belgian desserts",
    "Custard desserts",
    ...
  ],
  "templates": [
    "Short description",
    "Use American English",
    "Use mdy dates",
    ...
  ],
  "num_inlinks": 70,
  "redirects": [
    "Éclair",
    "Chocolate eclair",
    "Eclairs",
    ...
  ]
}

from datasets import load_dataset

# Load the dataset in streaming mode to avoid downloading it all at once
dataset = load_dataset("singletongue/wikipedia-paragraphs", name="enwiki-20260607-v1.2.1", split="train", streaming=True)

# Define a function to filter for relevant article pages
def filter_example(example):
    # Keep only article pages
    if example["page_type"] != "article":
        return False
    
    # Exclude articles that are just lists
    if example["title"].startswith("List of"):
        return False
    
    # Exclude disambiguation pages
    if "Disambiguation" in example["templates"]:
        return False
    
    return True

dataset = dataset.filter(filter_example)

# Define a function to extract and combine all paragraphs from the lead section
def get_lead_paragraph_text(example):
    lead_paragraphs = []
    for text, section in zip(example["paragraph_texts"], example["paragraph_sections"]):
        if section == "__LEAD__":
            lead_paragraphs.append(text)
    
    # Join the paragraphs into a single string, replacing newlines with spaces
    return {"lead_paragraph_text": " ".join(lead_paragraphs).replace("\n", " ")}

dataset = dataset.map(get_lead_paragraph_text)

# Take the first 100 articles and save their lead paragraphs to a file
with open("corpus.txt", "w") as fo:
    for example in dataset.take(100):
        print(example["lead_paragraph_text"], file=fo)