aipracticecafe/UniDic-tdmelodic

tdmelodic Pre-computed Accents Dataset

This repository provides pre-computed, inference-ready CSV files generated by tdmelodic (Tokyo Dialect MELOdic accent DICtionary generator) mapping over the NEologd vocabulary.

Generating these files locally requires running neural network inference (tdmelodic-convert), which typically takes several hours to complete depending on the hardware. We have pre-generated these dictionary files and made them available here to eliminate the setup and compute time.

Provided Files

• —`tdmelodic.csv`: The recommended dictionary file. It contains the inferred accents with unigram costs (word occurrence probabilities) adjusted using tdmelodic-modify-unigram-cost.
• —`tdmelodic_original.csv`: The raw output generated directly by tdmelodic-convert before applying unigram cost post-processing.

How to Use with MeCab & UniDic

From tdmelodic docs

To integrate these accents into your MeCab analyzer, you can merge tdmelodic.csv into the source files of UniDic (unidic-mecab_kana-accent-2.1.2_src) and compile the dictionary.

1. Download and Extract UniDic Source

First, download the UniDic source archive (unidic-mecab_kana-accent-2.1.2_src.zip) from official sources. Move it to your workspace and extract it:

# Define your workspace and unzip UniDic
WORKDIR=/path/to/your/workspace
cd ${WORKDIR}

unzip unidic-mecab_kana-accent-2.1.2_src.zip

2. Prepare tdmelodic.csv

Download the tdmelodic.csv file from this repository and place it in your workspace directory ${WORKDIR}.

3. Merge the Dictionaries

Navigate to the extracted UniDic directory, back up the default vocabulary (lex.csv), and append the pre-computed tdmelodic accents:

cd ${WORKDIR}/unidic-mecab_kana-accent-2.1.2_src

# Backup UniDic's original vocabulary
cp lex.csv lex_bak.csv

# Append the pre-computed tdmelodic dictionary
cat ${WORKDIR}/tdmelodic.csv >> lex.csv

4. Compile and Install

Configure, compile, and install the dictionary onto your system.

Note: Writing to /usr/lib requires administrator privileges. Ensure you run the make install command using sudo.

./configure --with-dicdir=`mecab-config --dicdir`/tdmelodic
make

# Install with root privileges
sudo make install

Verification & Examples

You can now parse Japanese sentences and retrieve accent patterns with MeCab.

CLI Usage

To analyze a sentence, specify the tdmelodic dictionary path:

echo 一昔前は人工知能のプログラミング言語といえばCommon LispやPrologだった。 | \
    mecab -d `mecab-config --dicdir`/tdmelodic/

Expected Output:

一昔	ヒトムカシ	ヒトムカシ	一昔	名詞-普通名詞-一般			2,3
前	マエ	マエ	前	名詞-普通名詞-副詞可能			1
は	ワ	ハ	は	助詞-係助詞
人工知能	ジ[ンコーチ]ノー	ジンコウチノウ	人工知能	名詞-固有名詞-一般			@
の	ノ	ノ	の	助詞-格助詞
プログラミング言語	プ[ログラミングゲ]ンゴ	プログラミングゲンゴ	プログラミング言語	名詞-固有名詞-一般			@
と	ト	ト	と	助詞-格助詞
いえ	イエ	イウ	言う	動詞-一般	五段-ワア行	仮定形-一般	0
ば	バ	バ	ば	助詞-接続助詞
Common Lisp	コ[モンリ]スプ	コモンリスプ	Common Lisp	名詞-固有名詞-一般			@
...

Python Usage (mecab-python3)

When using MeCab in Python, specify both the compiled dictionary path (-d) and your system's configuration file (-r, which is typically located at /etc/mecabrc in Ubuntu/Debian environments):

import subprocess
import MeCab

# 1. Retrieve the dictionary directory path
dicdir = subprocess.check_output(["mecab-config", "--dicdir"]).decode("utf-8").strip()
tdmelodic_path = f"{dicdir}/tdmelodic/"

# 2. Specify the system mecabrc path
mecabrc_path = "/etc/mecabrc"

# 3. Initialize MeCab and analyze
tagger = MeCab.Tagger(f"-d {tdmelodic_path} -r {mecabrc_path}")

text = "一昔前は人工知能のプログラミング言語といえばCommon LispやPrologだった。"
print(tagger.parse(text))

License & Credits

• —tdmelodic: Developed by PKSHA Technology. Licensed under the BSD-3-Clause license.
• —NEologd: Developed by Toshinori Sato. Licensed under the Apache License 2.0.
• —UniDic: Created by the National Institute for Japanese Language and Linguistics (NINJAL).

Please adhere to the respective licenses of tdmelodic, NEologd, and UniDic when using these pre-computed files.