xqt/jlpt_n5_vocabulary

Jisho JLPT-N5 Relational Dataset

This dataset provides a comprehensive and highly normalized collection of Japanese vocabulary for the JLPT-N5 level, scraped from Jisho.org. Unlike flat datasets, this version uses a relational schema to separate headwords, metadata (JLPT/WaniKani levels), and individual English definitions.

🏗 Scraper Architecture

The data was generated using a custom Python scraper following a robust state-machine logic.

Logic Highlights:

• —Normalization: Every English sense is a unique row with a specific meaning_no.
• —Integrity: Every entry is assigned a Version 4 UUID for reliable cross-referencing.
• —Resilience: Implements rate-limiting and session-based retries to ensure high data quality.

📂 Dataset Structure

The dataset consists of three primary relational tables:

1. raw_words.csv (Headwords)

2. raw_meta.csv (Linguistic Context)

3. raw_meanings.csv (Definitions)

🛠 How to Use

Since the data is normalized, you can use Pandas to merge the tables into a flat view for training or analysis:

import pandas as pd

words = pd.read_csv('raw_words.csv')
meta = pd.read_csv('raw_meta.csv')
meanings = pd.read_csv('raw_meanings.csv')

# Merge into a single dataframe
df = words.merge(meta, on='word_id').merge(meanings, on='word_id')
print(df.head())

📊 Dataset Stats

• —Total Headwords: ~700-800
• —Total Definitions: ~2,500+
• —Source: Jisho.org API
• —Level: JLPT N5 (Beginner)

📜 License & Acknowledgments

This dataset is distributed under the MIT License. All linguistic data belongs to Jisho.org and its respective contributors.