Xav007/dyslexia-accessibility-nlp

title: Dyslexia Accessibility NLP emoji: 📖 colorFrom: blue colorTo: indigo sdk: gradio appfile: app/main.py pinned: false sdkversion: 6.18.0 ---

Dyslexia Accessibility NLP

A multi-model deep learning pipeline for dyslexia screening from handwriting images. Three heterogeneous models — a scikit-learn MLP letter classifier, a PyTorch CNN reversal detector, and a PyTorch Bidirectional LSTM sequence anomaly detector — are fused via a clinically motivated weighted ensemble and served through a Flask web application with structured PDF reporting.

This project is framed as a research capstone on multi-model fusion for social impact, not a production tool. Every architectural and mathematical decision is motivated and documented below.

Project Structure

dyslexia-accessibility-nlp/
│
├── beta versions/                  ← Original iterative development history
│   ├── Letter_Classification/        · Scratch MLP (NumPy/Numba), GridSearch,
│   │   ├── scripts/                    output predictions at 10k/30k/88.8k samples
│   │   └── output/
│   ├── Dyslexic_Detection/           · TF/Keras CNN training notebook,
│   │   └── testing_tf.ipynb            checkpoint .h5 models
│   ├── nlp_module.py                 · Original incomplete LSTM module
│   └── README.md
│
├── data/                           ← Data loading, preprocessing, augmentation
│   ├── preprocessing.py              · EMNIST loader (orientation fix),
│   │                                   stratified splits, StandardScaler
│   ├── augmentation.py               · torchvision transforms + PyTorch DataLoader
│   └── nlp_data_generator.py         · Synthetic sequence generator with
│                                       MLP confusion noise (domain adaptation)
│
├── models/                         ← Model definitions and training scripts
│   ├── mlp_classifier.py             · 3-layer MLP (512→256→128), Adam,
│   │                                   sklearn early stopping
│   ├── cnn_classifier.py             · 3-block Conv2D CNN, BatchNorm, Dropout,
│   │                                   GlobalAvgPool, PyTorch AMP (GPU)
│   ├── nlp_sequence.py               · Bidirectional 2-layer LSTM,
│   │                                   PyTorch AMP (GPU)
│   └── ensemble.py                   · Strong-binary sliding window ensemble
│                                       + analytical NLP pattern scorer
│
├── pipeline/                       ← Inference and report generation
│   ├── character_extraction.py       · Adaptive threshold OpenCV pipeline
│   │                                   with Otsu fallback
│   ├── inference.py                  · Unified entry point — all 3 models
│   │                                   + analytical/LSTM NLP blend
│   └── report_generator.py           · ReportLab Platypus PDF report
│
├── evaluation/
│   └── benchmark.py                  · F1, AUC, confusion matrix,
│                                       reversal-pair analysis (b/d, p/q, n/u, m/w)
│
├── app/
|   ├── images/                       · some images to test
|   ├── reports/                      · some sample reports on the test images
│   ├── main.py                       · Flask app — UUID sessions, MIME
│   │                                   validation, auto-expiring PDF reports
│   ├── models/                       · Trained model files (gitignored)
│   │   ├── mlp_model.pkl
│   │   ├── mlp_scaler.pkl
│   │   ├── pattern_classifier.pt
│   │   └── sequence_anomaly.pt
│   └── templates/
│       └── index.html                · Single-page frontend
│
├── output/                         ← Runtime output (gitignored)
│   ├── characters/                   · Temp per-session character crops
│   ├── reports/                      · Generated PDFs (auto-deleted, 5 min)
│   └── benchmarks/                   · JSON benchmark results
│
├── data/raw/                       ← Raw datasets (gitignored)
│   ├── emnist-letters-train.csv
│   ├── emnist-letters-test.csv
│   └── Gambo/
│       ├── Train/{Normal,Reversal}/
│       └── Test/{Normal,Reversal}/
│
├── gpu_config.py                   ← PyTorch GPU detection + logging
├── config.py                       ← Single source of truth: all paths,
│                                     hyperparameters, thresholds
├── train_all.py                    ← Master training orchestrator
├── requirements.txt
├── .gitignore
└── README.md

Inference Pipeline

Upload image
    │
    ▼
character_extraction.py
  Greyscale → Gaussian denoise
  → Adaptive Gaussian threshold (handles uneven lighting)
  → Morphological closing (reconnects broken strokes)
  → Contour filter: area-ratio [0.00005, 0.20] + aspect [0.05, 5.0]
  → Otsu fallback if adaptive finds nothing
  → Sort left-to-right (reading order)
  → Resize → 28×28 (MLP) and 64×64 (CNN)
    │
    ▼  (per character, batched)
MLP (sklearn, CPU)
  StandardScaler → 3-layer MLP → letter (A–Z) + softmax confidence
    │
CNN (PyTorch, GPU)
  Conv2D × 3 blocks → GlobalAvgPool → sigmoid → reversal probability [0,1]
    │
    ▼
Quality Gate (pipeline/inference.py)
  Check 1 — rare letter dominance:
    rare_ratio = count(z,x,q,j,w,v) / n
    if rare_ratio > 0.55 → Inconclusive
    (noise/background regions produce these letters; real text never dominates with them)
  Check 2 — raw CNN reversal ceiling:
    if mean(cnn_probs) > 0.65 → Inconclusive
    (worst-case genuine dyslexic text stays below 0.65; noise regions exceed it)
    │
    ▼
NLP component (hybrid)
  Analytical score:
    strong_count (CNN ≥ 0.85) / max(n × 0.08, 5)
    — clinically normalised against 8% expected reversal rate
    — does not saturate from cursive noise (50–80% CNN)
      vs true reversals (90–100% CNN)
  LSTM blend (when retrained on noise-aware data):
    if 0.03 < lstm_output < 0.97:
        nlp = 0.70 × analytical + 0.30 × lstm
    else:
        nlp = analytical only   ← saturation check gates broken LSTM output
    │
    ▼
Ensemble (models/ensemble.py)
  CNN component — strong-binary sliding window:
    binary = (reversal_probs >= 0.85)
    window = 15% of n, clamped to [5, 20] characters
    local_weight = clip(0.20 + (n − 10) × 0.007, 0.20, 0.80)
    cnn_component = (1 − local_w) × global_strong_rate
                  + local_w × sliding_peak

  ensemble_score = 0.55 × cnn_component
                 + 0.40 × nlp_component
                 + 0.05 × mlp_uncertainty

  threshold = 0.40
    │
    ▼
DiagnosisResult → JSON response + PDF report

Setup

# 1. Create and activate virtual environment
python -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate

# 2. Install dependencies
pip install -r requirements.txt

# PyTorch with CUDA (recommended — CPU fallback works but CNN/NLP will be slow):
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121

# 3. Place datasets (see Datasets section)

# 4. Train models
python train_all.py          # all three
python train_all.py 1        # MLP only  (sklearn, CPU,  ~5 min)
python train_all.py 2        # CNN only  (PyTorch, GPU, ~30–60 min)
python train_all.py 3        # NLP only  (PyTorch, GPU, ~10 min)
python train_all.py 2 3      # CNN + NLP (skip MLP if already trained)

# Optional flags
python train_all.py 2 3 --smoke-test          # tiny data slice, fast validation
python train_all.py 2 3 --no-mixed-precision  # disable AMP if you see NaN losses
python train_all.py 2 3 --skip-eval           # skip benchmark after training

# 5. Run the app
python app/main.py
# → http://localhost:5000

Datasets

EMNIST note: The CSV format stores images rotated 90° clockwise and horizontally mirrored. preprocessing.py undoes both transforms (np.rot90(k=3) + np.fliplr) before any training — without this the MLP trains on visually incorrect characters.

NLP note: Delete data/raw/sequence_data.txt and run python train_all.py 3 to regenerate training data with domain adaptation noise before retraining the LSTM.

Model Performance

Benchmarked on held-out test splits. Full metrics including per-class F1 and confusion matrices in output/benchmarks/.

Reversal-pair confusion is tracked explicitly in the MLP benchmark — the b/d, p/q, n/u, and m/w pairs are the most clinically significant confusions and are reported separately from overall accuracy.

Key Design Decisions

Why strong-binary reversal threshold (CNN ≥ 0.85)?

The CNN's raw reversal probability for cursive strokes, ink bleed, and touching letters consistently lands between 50–80%. Genuine letter reversals (b→d, p→q) consistently land at 90–100%. A hard threshold at 0.85 separates signal from noise without any learned parameter — it is a principled operating point derived from the CNN's ROC curve.

Using a global mean across all characters dilutes genuine reversal clusters in long paragraphs. A writer who reverses 20 out of 100 characters scores a mean of 20% — clinically significant, but the mean hides it. The sliding window captures the densest local burst and blends it with the global rate, where the blend weight shifts toward local as text length grows (n=10 → localw=0.20, n=100 → localw=0.80). This is motivated by clinical literature describing dyslexic errors as localised clusters rather than uniformly distributed noise.

Why the analytical NLP score instead of raw LSTM output?

The LSTM trained on clean synthetic sequences but receives the MLP's noisy letter predictions at inference. The MLP misclassifies ~10% of characters — within visually similar groups (c/e, i/j, u/v, n/m). To the LSTM, every real input looks anomalous regardless of dyslexia status, saturating near 100% for all inputs.

The analytical score addresses this by counting only strong reversals (CNN ≥ 0.85) normalised against clinical expectation (8% of characters are strongly reversed in diagnosed dyslexic writers, per Isa et al. 2019). It cannot saturate from cursive noise and gives a meaningful zero for clean writing.

The LSTM remains in the architecture and contributes 30% when its output is in a valid range (0.03–0.97), gated by a saturation check. It will become the dominant NLP signal once retrained on noise-aware data.

Why domain adaptation in NLP training data?

This is a training-inference distribution mismatch. The fix (_simulate_mlp_noise()) applies structured MLP confusion substitutions to both training classes before dyslexic transformations are added to the anomalous class. This ensures the LSTM sees the same noise floor at training time as at inference, forcing it to learn the boundary between baseline MLP noise (normal) and dyslexic patterns above that baseline — the exact distinction it needs to make.

Why ensemble fusion over a single model?

Each model captures a distinct and complementary signal:

No single modality is sufficient. A writer who reverses letters cleanly (high MLP confidence, high CNN reversal) would be missed by a sequence-only model. A writer with poor handwriting quality (low MLP confidence) but no reversals would be over-flagged by a CNN-only model. The ensemble with domain-motivated weights handles both cases.

Why not horizontal flip augmentation for the CNN?

Deliberately excluded. A horizontally flipped 'b' is a 'd' — which is exactly the reversal pattern being detected. Including it as augmentation would teach the model that both orientations are equivalent, destroying its ability to detect reversals. This is a domain-specific augmentation choice motivated by the nature of the classification task.

Why a quality gate before the ensemble?

When character extraction fails — due to a noisy image, extreme angle, faint ink, or cluttered background — the downstream models process garbage and produce confidently wrong outputs. The ensemble has no way to know the input was invalid.

Two independent checks catch this before any scoring happens:

Rare letter dominance: Background noise and stroke fragments get misclassified as z, x, q, j, w, v — letters with distinctive shapes that match common noise patterns. These letters make up under 3% of normal English text. If they exceed 55% of predictions, the extractor grabbed non-character regions. This check is deliberately inverted from checking for common letter presence — short real words like "KITE FAMILY" legitimately contain few common letters (e, t, a, o, i, n...) but will never be dominated by rare ones.

Raw reversal ceiling: Even worst-case dyslexic short text — 4 of 10 characters genuinely reversed plus moderate CNN responses on visually ambiguous letters — stays below 0.65 raw mean. Noise regions consistently exceed this. The threshold provides headroom for genuine edge cases while blocking haywire extractions.

Both thresholds are set conservatively to avoid false Inconclusive results on legitimate images, and both are configurable in QualityGateConfig in config.py. ---

Key Improvements Over Beta Versions

Referenced Research

• —Alqahtani, N. D., et al. (2023). "Detection of Dyslexia Through Images of Handwriting using Hybrid AI Approach." International Journal of Advanced Computer Science and Applications (IJACSA).
• —Alqahtani, N. D., et al. (2023). "Deep Learning Applications for Dyslexia Prediction." Applied Sciences.
• —Isa, I. S., et al. (2019). "Automated Detection of Dyslexia Symptom Based on Handwriting Image for Primary School Children." Procedia Computer Science.
• —Cohen, G., et al. (2017). "EMNIST: an extension of MNIST to handwritten letters." arXiv:1702.05373.

Disclaimer

This tool is a screening aid for educational and research purposes only. It does not constitute a medical or psychological diagnosis. Always consult a qualified educational psychologist for clinical assessment.