Thoria/mandarin-most-common-words-tr-en

Mandarin Most Common Words (TR-EN)

Overview

The Mandarin Most Common Words (TR-EN) dataset is a comprehensive trilingual vocabulary resource designed for learners of Mandarin Chinese. It provides translations and practical examples in both Turkish and English, making it highly useful for bilingual education, language learning apps, and linguistic analysis.

This dataset was created by Stephanie Liu and Kamil Murat Yilmaz.

Dataset Content

The dataset contains 1,143 commonly used Mandarin Chinese words and phrases, categorized into 25 everyday topics. Each entry includes the Chinese characters (Hanzi), pronunciation (Pinyin), translations, and contextual example sentences in all three languages.

Features (Columns)

The CSV file consists of the following 12 columns:

• —hanzi: The word/phrase written in Simplified Chinese characters.
• —pinyin: The phonetic transcription of the Chinese characters.
• —tr: The Turkish translation of the word.
• —en: The English translation of the word.
• —example_zh: An example sentence using the word in Chinese.
• —example_tr: The Turkish translation of the example sentence.
• —example_en: The English translation of the example sentence.
• —category: The thematic category the word belongs to.
• —frequency_zipf: The Zipf frequency of the word in modern Chinese (see below).
• —zh_audio: JSON with all TTS audio URLs for the Chinese word and example sentence (see Audio).
• —en_audio: JSON with TTS audio URLs for the English word and example sentence.
• —tr_audio: JSON with TTS audio URLs for the Turkish word and example sentence.

Zipf Frequency

The frequency_zipf column reports each word's Zipf frequency, a normalized, human-readable scale defined as log10(occurrences per billion words) + 3. Values typically fall in the range 0–8:

Because the scale is logarithmic, each +1 point ≈ 10× more frequent. A word at Zipf 6 is ~1000× more common than one at Zipf 3.

Frequencies are computed via the `wordfreq` library, whose Simplified Chinese wordlist aggregates several real-world corpora — including SUBTLEX-CH (film & TV subtitles), OpenSubtitles, Wikipedia, and microblog data — giving a balanced view of both spoken-register and written-register usage.

To regenerate the column after editing the dataset:

uv sync
uv run python annotate_zipf.py

Audio

Each row is paired with up to 12 TTS audio clips — every word and example sentence has a normal-speed and a slow-speed recording, for each of the three languages. The files are hosted in the HuggingFace bucket `Thoria/TTS-UMAY` and addressable via URLs of the form:

https://huggingface.co/buckets/Thoria/TTS-UMAY/resolve/{folder}/{speed}/row_{index}.wav

where folder ∈ {hanzi, en, tr, example_zh, example_en, example_tr}, speed ∈ {normal, slow}, and index is the 0-based CSV row number.

All six URLs for a given language are bundled into a single JSON column (zh_audio, en_audio, tr_audio) with the shape language → part → speed:

{
  "sentence": {
    "normal": "https://huggingface.co/buckets/Thoria/TTS-UMAY/resolve/example_zh/normal/row_0.wav",
    "slow":   "https://huggingface.co/buckets/Thoria/TTS-UMAY/resolve/example_zh/slow/row_0.wav"
  },
  "word": {
    "normal": "https://huggingface.co/buckets/Thoria/TTS-UMAY/resolve/hanzi/normal/row_0.wav",
    "slow":   "https://huggingface.co/buckets/Thoria/TTS-UMAY/resolve/hanzi/slow/row_0.wav"
  }
}

The Turkish column (tr_audio) is pre-wired in the schema but the corresponding audio files are not yet available on the bucket.

Listening from a browser

The /resolve/ URLs above return the raw WAV file — perfect for embedding in <audio> tags, feeding into loaders like librosa / torchaudio, or piping into curl/wget. A direct /resolve/ request from the browser will trigger a download rather than open a player.

To preview a single clip in the HuggingFace web UI (inline player, file metadata, folder navigation), swap /resolve/ for /tree/ in the URL, e.g.:

https://huggingface.co/buckets/Thoria/TTS-UMAY/tree/hanzi/normal/row_0.wav

Or browse the whole collection visually here: huggingface.co/buckets/Thoria/TTS-UMAY

To regenerate the audio columns (e.g. after changing the bucket host):

uv run python annotate_audio.py

The bucket root is configurable via the DEFAULT_BUCKET constant in `annotate_audio.py`, or by passing bucket=... to audio_path() if you import the module.

Categories

The vocabulary is organized into the following 25 practical categories:

• —Greetings
• —Self-introduction
• —Places
• —Jobs
• —Shopping
• —Activities
• —Food
• —Family
• —Colors
• —Fruits
• —Dates & Time
• —Numbers
• —Emotions
• —Sports
• —Descriptive Words
• —Countries
• —Animals
• —Nature & Weather
• —Household Items
• —Clothing
• —Hotel
• —Directions
• —Taxi/Transportation
• —Airport
• —Apps/Technology

Anki Flashcards

The dataset ships with pre-built Anki .apkg decks in `anki_cards/` — one per direction. Double-click any file in Finder to install.

To regenerate (or rebuild after editing the CSV), run `csv_to_apkg.py`:

# All 6 directions → anki_cards/*.apkg
uv run python csv_to_apkg.py --mode all

# Single direction
uv run python csv_to_apkg.py --mode en_tr

# Several, to a custom folder
uv run python csv_to_apkg.py --mode zh_tr tr_zh en_zh --outdir ./decks

By default the .apkg embeds bucket URLs as <audio> sources and falls back to local files when present (hybrid mode). For fully offline, self-contained decks first run `download_audio_anki.py`, then:

uv run python csv_to_apkg.py --mode all --audio offline --bundle-media

Available Modes

Potential Use Cases

• —Language Learning: Creating flashcards, quizzes, and vocabulary lists for students learning Mandarin via Turkish or English.
• —NLP & Linguistics: Training multilingual translation models or analyzing cross-linguistic sentence structures.
• —Application Development: Acting as a foundational database for vocabulary apps or phrasebooks.

Credits

Original creators: Stephanie Liu and Kamil Murat Yılmaz

Contributors: see commit history

Language: English, Turkish, Mandarin Chinese

License

This repository is dual-licensed to keep the reusable code and the curated dataset cleanly separated:

What this means

• —Code — commercial use, modification, and redistribution are permitted under the MIT License. You must preserve the copyright notice and the MIT permission notice when redistributing.
• —Dataset — commercial use, modification, and redistribution are permitted under CC BY 4.0. You must credit the creators and indicate if you modified the dataset.

Attribution

When redistributing or citing the dataset, please include:

Mandarin Most Common Words (TR-EN) — created by Stephanie Liu, Kamil Murat Yılmaz, and contributors. Licensed under CC BY 4.0. https://huggingface.co/datasets/Thoria/mandarin-most-common-words-tr-en

BibTeX:

@dataset{liu_yilmaz_mandarin_tr_en_2026,
  author    = {Liu, Stephanie and Yılmaz, Kamil Murat},
  title     = {Mandarin Most Common Words (TR-EN)},
  year      = {2026},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/datasets/Thoria/mandarin-most-common-words-tr-en},
  license   = {CC BY 4.0}
}