undertheseanlp/sentence-segmentation-1

Sentence Segmentation

Test set for evaluating and improving Vietnamese sentence boundary detection (sent_tokenize) in underthesea.

Problem

The current PunktSentenceTokenizer in underthesea fails on several Vietnamese-specific patterns, primarily in legal text where article titles are merged with sentence bodies without punctuation boundaries.

Current Results

Key Issues

1. 1.Title-content merge (38 cases, 0% accuracy): Legal article titles like "Tội trốn thuế" are followed by sentence body "Người nào thực hiện..." without punctuation. sent_tokenize fails to detect this boundary.

1. 1.Repeated title (13 cases, 0% accuracy): Pattern "X X là..." where the title is repeated as the subject of a definition. E.g., "Hợp đồng mượn tài sản Hợp đồng mượn tài sản là..."

1. 1.Ellipsis handling (0% accuracy): "..." mid-sentence causes incorrect split.

1. 1.Numeric periods (0% accuracy): Periods in numbers like "1.500.000" can cause false boundaries.

Trained Punkt Model Results

Trained NLTK PunktTrainer on Vietnamese text from 4 sources (Wikipedia, news, books, legal documents). The trained model fixes punctuation-related issues but cannot address structural patterns (title-content merge).

Improvements: ellipsis handling (+1), numeric period handling (+1) Regressions: multi_sentence (-2) — trade-off from better ellipsis handling (... followed by new sentence) and quote tokenization differences.

Conclusion: Punkt (trained or not) cannot solve title-content merge (51/130 failures = 39% of test set) because these require structural understanding beyond punctuation disambiguation. A different approach is needed for this category.

Files

• —test_cases.json — 130 test cases with input, expected output, category, and domain
• —evaluate.py — Evaluation script (--improved flag for trained model)
• —eval_results.json — Detailed evaluation results
• —train_punkt.py — Fetch data + train Punkt model
• —punkt_params_trained.json — Trained model parameters (672 abbreviations, 378 sentence starters, 3264 collocations)
• —sent_tokenize.py — Tokenizer using trained model

Test Case Format

{
  "id": "vlc-6200",
  "input": "Tội ngược đãi tù binh , hàng binh Người nào ngược đãi tù binh ...",
  "expected": [
    "Tội ngược đãi tù binh , hàng binh",
    "Người nào ngược đãi tù binh , hàng binh , thì bị phạt ..."
  ],
  "category": "title_content_merge",
  "domain": "legal",
  "issue": "Title merged with sentence body without boundary"
}

Usage

# Run baseline evaluation (underthesea)
python evaluate.py
python evaluate.py -v

# Run trained model evaluation
python evaluate.py --improved -v

# Train Punkt model from scratch
python train_punkt.py

Data Source

Test cases derived from undertheseanlp/UDD-1 across 5 domains: legal (VLC), news (UVN), Wikipedia (UVW), fiction (UVB-F), non-fiction (UVB-N).

Categories