Youngmeen/language-learning-analytics

1---2title: Language Learning Analytics3emoji: 📊4colorFrom: blue5colorTo: green6sdk: docker7app_port: 78608pinned: false9---10 11# Linguistic Analytics Service12 13FastAPI microservice computing the SPEC §7 **text-based** linguistic metrics on a learner's14session production. Called by the Next.js app after a session ends; writes results to15`analytics_results` in Supabase (service-role key — the only writer by design).16 17- `POST /analyze {"session_id": "..."}` — `Authorization: Bearer $ANALYTICS_SERVICE_SECRET`18- `GET /health`19 20Env vars: `SUPABASE_URL`, `SUPABASE_SERVICE_ROLE_KEY`, `ANALYTICS_SERVICE_SECRET`.21 22## Metrics — version `text-1.0`23 24Computed on **learner turns only**, tokenized/parsed with spaCy small models25(`en_core_web_sm`, `ko_core_news_sm`, `ja_core_news_sm`, `zh_core_web_sm`,26`fr_core_news_sm`, `de_core_news_sm`). Small models are a deliberate v1 tradeoff for27hosting cost; an upgrade (e.g. Stanza) bumps `metrics_version` so results stay comparable28within a version. "Words" = alphabetic tokens, lowercased.29 30### volume31| key | definition |32|---|---|33| token_count | alphabetic tokens across learner turns |34| utterance_count | learner turns |35 36### lexical_diversity37| key | definition | null when |38|---|---|---|39| ttr | types / tokens | no tokens |40| mattr | moving-average TTR, window 50 | < 50 tokens |41| mtld | bidirectional MTLD, factor TTR threshold 0.72 (McCarthy & Jarvis 2010) | < 50 tokens |42| hdd | HD-D with sample size 42 (vocd-D analogue, McCarthy & Jarvis 2007) | < 42 tokens |43 44### lexical_sophistication (wordfreq Zipf scale, per target language)45| key | definition |46|---|---|47| mean_zipf | mean Zipf frequency of tokens known to wordfreq |48| low_freq_ratio | share of known tokens with Zipf < 4.0 |49| advanced_ratio | share of known tokens with Zipf < 3.0 (the "academic/advanced" proxy — language-neutral; a curated academic word list exists only for English and is deferred) |50| unknown_ratio | share of all tokens unknown to wordfreq (typos, names) — excluded from the ratios above |51| band_common / band_mid / band_rare | known-token shares with Zipf ≥ 5 / 4–5 / < 4 |52 53### syntax (dependency-parse approximations)54| key | definition |55|---|---|56| mlu_words | mean words per utterance (word-based, not morpheme-based — a documented simplification, esp. for zh/ja) |57| mean_t_unit_length | words / main clauses; main clauses = sentence roots + verbal conjuncts of the root |58| clauses_per_utterance | (main + subordinate clauses) / utterances |59| subordination_ratio | subordinate / all clauses; subordinate labels: advcl, ccomp, xcomp, acl, relcl, acl:relcl, csubj(:pass) |60| mean_parse_depth / max_parse_depth | dependency tree depth per sentence (root = 1) |61 62### repetition63| key | definition |64|---|---|65| immediate_repetition_count | immediately repeated tokens ("the the") within an utterance |66| repeated_bigram_ratio | share of within-utterance bigram instances that are duplicates |67 68### repair (typing-mode heuristic — limited until speech mode adds timing)69| key | definition |70|---|---|71| correction_marker_count | chat-convention self-corrections (`*word`) |72 73## Development74 75```bash76uv sync77uv pip install https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.8.0/en_core_web_sm-3.8.0-py3-none-any.whl78uv run pytest          # metric tests are hand-computed known-input/expected-output79uv run uvicorn app.main:app --port 786080```81