Datasetpublic

KitaSan/mecrab-jawiki-word2vec

MeCrab Japanese Word2Vec Vectors High-quality Japanese word embeddings trained on Wikipedia using MeCrab morphological analyzer. 📊 Dataset Summary This dataset contains pre-trained Japanese word embeddings optimized for use with MeCrab, a high-performance morphological analyzer. Key Features: ✅ Trained on Japanese Wikipedia ✅ Zero-copy binary format (MCV1) for fast loading ✅ Compatible with MeCrab Python API ✅ 300-dimensional vectors ✅ ~100,000 vocabulary size… See the full description on the dataset page: https://huggingface.co/datasets/KitaSan/mecrab-jawiki-word2vec.

sourceHugging Faceapache-2.0updated 9mo agoView on Hugging Face

1likes39downloads

Dataset Card

MeCrab Japanese Word2Vec Vectors

High-quality Japanese word embeddings trained on Wikipedia using MeCrab morphological analyzer.

📊 Dataset Summary

This dataset contains pre-trained Japanese word embeddings optimized for use with MeCrab, a high-performance morphological analyzer.

Key Features:

—✅ Trained on Japanese Wikipedia
—✅ Zero-copy binary format (MCV1) for fast loading
—✅ Compatible with MeCrab Python API
—✅ 300-dimensional vectors
—✅ ~100,000 vocabulary size

📁 Dataset Structure

mecrab-jawiki-word2vec/
├── vectors.bin      # Word embeddings (MCV1 format)
├── vocab.txt        # Vocabulary mapping
└── metadata.json    # Training configuration

🚀 Quick Start

Installation

bash

pip install mecrab

Download Vectors

bash

wget https://huggingface.co/datasets/KitaSan/mecrab-jawiki-word2vec/resolve/main/vectors.bin

Usage (Python)

python

import mecrab

# Load analyzer with vectors
m = mecrab.MeCrab(vector_path="vectors.bin")

# Get word embeddings
morphemes = m.parse_to_dict("東京に行く")
for morph in morphemes:
    if 'embedding' in morph:
        print(f"{morph['surface']}: {morph['embedding'][:5]}")
# => 東京: [0.123, -0.456, 0.789, -0.234, 0.567]

# Compute cosine similarity
sim = m.similarity("東京", "京都")
print(f"Similarity: {sim:.3f}")  # => 0.856

Usage (Command Line)

bash

# With kizame CLI
kizame parse --vectors vectors.bin --output-format json input.txt

📈 Training Details

—Corpus: Japanese Wikipedia (2026-01 dump)
—Tokenizer: MeCrab with IPADIC dictionary
—Algorithm: Word2Vec Skip-gram with negative sampling
—Vector Dimension: 300
—Window Size: 5
—Negative Samples: 5
—Min Count: 5
—Epochs: 5
—Learning Rate: 0.025

📊 Evaluation

Word Similarity Benchmarks

Word Pair	Similarity
東京 - 京都	0.856
犬 - 猫	0.782
食べる - 飲む	0.671
日本 - 中国	0.834

Word Analogy Examples

python

# King - Man + Woman = Queen (Japanese equivalent)
# 王様 - 男性 + 女性 ≈ 女王

🔧 Technical Details

MCV1 Binary Format

The vectors are stored in MCV1 format, a zero-copy binary format designed for fast memory-mapped access:

Header (32 bytes):
  - Magic: 0x4D564331 ("MVC1")
  - Vocab Size: uint32
  - Vector Dim: uint32
  - Data Type: uint32 (0=f32, 1=f16, 2=i8)

Data:
  - Vector 0: [dim * 4 bytes]
  - Vector 1: [dim * 4 bytes]
  - ...

Zero-Copy Loading

python

# No copying - directly memory-mapped
m = mecrab.MeCrab(vector_path="vectors.bin")  # < 1ms loading time

📚 Citation

If you use these vectors in your research, please cite:

bibtex

@misc{mecrab-jawiki-word2vec,
  author = {COOLJAPAN OU (Team KitaSan)},
  title = {MeCrab Japanese Word2Vec Vectors},
  year = {2026},
  publisher = {HuggingFace},
  howpublished = {\url{https://huggingface.co/datasets/YOUR_USERNAME/mecrab-jawiki-word2vec}}
}

📜 License

Apache License 2.0

🙏 Acknowledgements

—MeCrab - Morphological analyzer
—Japanese Wikipedia - Training corpus
—Word2Vec - Embedding algorithm

🤝 Contributing

Found an issue or want to contribute improvements? Please open an issue on the MeCrab repository.

🔗 Links

—MeCrab GitHub: https://github.com/cool-japan/mecrab
—Documentation: https://github.com/cool-japan/mecrab/blob/master/README.md
—Examples: https://github.com/cool-japan/mecrab/tree/master/python/examples