Datasetpublic

ruvimx/UkrLM-social

UkrLM Social Corpus A curated corpus of Ukrainian-language text collected from public social media platforms, designed for language model pretraining and fine-tuning. This dataset is part of the UkrLM initiative — an open effort to build foundational NLP resources for the Ukrainian language. Overview Property Value Language Ukrainian (uk) Sources Telegram, Reddit License CC BY 4.0 Format Parquet Task Language Modeling Sources… See the full description on the dataset page: https://huggingface.co/datasets/ruvimx/UkrLM-social.

sourceHugging Facecc-by-4.0updated 5mo agoView on Hugging Face

0likes30downloads

Dataset Card

UkrLM Social Corpus

A curated corpus of Ukrainian-language text collected from public social media platforms, designed for language model pretraining and fine-tuning.

This dataset is part of the UkrLM initiative — an open effort to build foundational NLP resources for the Ukrainian language.

Overview

Property	Value
Language	Ukrainian (`uk`)
Sources	Telegram, Reddit
License	CC BY 4.0
Format	Parquet
Task	Language Modeling

Sources

Telegram — public Ukrainian-language channels covering analysis, culture, and community discussion.

Reddit — public posts and comments from Ukrainian-language and Ukraine-focused subreddits including r/ukraine, r/ukr, and others.

Processing Pipeline

Raw text was passed through a multi-stage cleaning pipeline before inclusion:

Language & quality filtering

—Ukrainian only — pre-filtered at collection time
—Minimum 15 characters, maximum 10 000 characters
—Minimum 35% alphabetic character ratio
—Minimum 5 Cyrillic characters per record
—Records consisting primarily of anonymization tokens are discarded

Noise removal

—Markdown stripped: **bold**, _italic_, ~~strikethrough~~, > quotes, headings, inline code
—Ad tails removed line-by-line from the bottom of posts (channel promotions, social links)
—Emoji-heavy lines removed (>50% emoji by character count)
—Separator lines removed: ————, ___, etc.
—HTML entities decoded: > → >, & → &, etc.

Anonymization

—Person names - <PERSON>
—Usernames - <USER>
—Phone numbers - <PHONE>
—Card numbers - <CARD>
—Email addresses - <EMAIL>
—URLs - <URL>

Deduplication

—Exact match deduplication on lowercased, whitespace-normalized text

Format

Each record is a flat object with the following fields:

json

{"text": "...", "source": "telegram", "channel": "nazva_kanalu", "date": "2025-08-05T16:19:38+00:00", "lang": "uk"}
{"text": "...", "source": "reddit", "subreddit": "ukraine", "date": "2024-11-12T10:00:00+00:00", "score": 42, "lang": "uk"}

Field	Description
`text`	Cleaned Ukrainian text
`source`	Origin platform: `telegram` or `reddit`
`channel`	Telegram channel identifier (telegram only)
`subreddit`	Subreddit name (reddit only)
`date`	ISO 8601 timestamp (may be empty)
`score`	Reddit upvote score (reddit only)
`lang`	Language code — always `uk`

Part of UkrLM

This dataset is one component of the broader UkrLM project, which aims to produce open Ukrainian-language datasets and eventually a trained language model.

Related datasets in this initiative:

—ruvimx/UkrLM-social — social corpus (this dataset)
—ruvimx/UkrLM-wiki — Ukrainian Wikipedia

License

This dataset is released under Creative Commons Attribution 4.0 International (CC BY 4.0) for research purposes.

Source platforms (Telegram, Reddit) retain their own terms of service over the original content.

Built with ❤️ for the Ukrainian NLP community.