israelkingz/wa-commerce-blindspot-eval

West African Micro-Commerce Blind Spot Eval

Author: Israel Olanrewaju Odeajo (israelkingz) Challenge: Fatima Fellowship — Technical Challenge (Blind Spots of Frontier Models) Model under test: `Qwen/Qwen2.5-3B-Instruct` (~3B open-weight) Scope: A 24-item diagnostic set (not a large-scale benchmark claim) + reproducible GPU eval + qualitative failure analysis

1. The blind spot (lived experience)

Title

Semantic inversion and high-stakes failure modes in informal West African micro-commerce and peer-to-peer financial directives

I am writing this from West African settlement practice I know firsthand: everyday buying and selling settled over Nigerian Pidgin (Naija) mixed with English, on rails like NIP, OPay, Moniepoint, and bank apps—where a merchant’s only safe proof is a credit on their own phone, not a buyer’s story, screenshot, or debit SMS.

In that world, critical financial meaning rides on:

• —Aspect markers: don (completed) vs wan / make I / I go (intent / future)
• —Idioms: hold body (wait / do not release), run am (execute transfer), credit alert don drop (settled), e never clear (not settled)
• —Proof asymmetry: a debit SMS on the buyer's phone is not a credit on the merchant's account

Standard English finance / safety benchmarks (FinQA-style QA, explicit fraud prompts, clear imperatives) rarely test these pragmatics. When small instruction models power support bots or agentic “release goods?” tools, they can treat intent or buyer-side debit as settlement, which is exactly how micro-merchants lose inventory to fake-alert and pressure scams.

This gap is underexplored because it sits at the intersection of dialectal aspect, settlement rail mechanics, and social-engineering urgency—not sentiment accuracy or generic multilingual MMLU.

2. Systematic evaluation

Model choice

Qwen2.5-3B-Instruct (within 0.6B–6B): one of the stronger sub-4B instruct models with multilingual pretraining. Failures here are more informative than failures on a weak baseline—they point to a representational / cultural gap, not merely “the model is too small to follow instructions.”

Diagnostic set (24 items — not a leaderboard claim)

This is a hand-built diagnostic to surface failure modes, not a statistically powered benchmark. Axes:

Dataset: `data/eval_dataset.json`

Protocol

1. 1.Load model with chat template; greedy decoding (do_sample=False).
2. 2.System prompt asks for short analysis plus structured STATE / ACTION / RISK lines.
3. 3.Automatic scores use strict/rough label match against a fixed gold vocabulary (see caveat below).
4. 4.Qualitative review in `results/human_notes.md` focuses on dangerous semantic errors, not synonym mismatches.

How to read the metrics (important)

Strict state accuracy is 0/24 under our gold label vocabulary. That number is useful but easy to over-interpret:

• —Many generations are near-miss paraphrases of a pending/uncertain state (INQUIRY, In Progress, UNKNOWN) while gold expects a canonical tag (PENDING_CONFIRMATION, NOT_SENT). That inflates “failure” on Axis A.
• —The fellowship-relevant failures are semantic inversions and under-flagged risk—e.g. item 3 emitting STATE: VERIFIED on a buyer debit-only claim, item 16 emitting CONFIRMED on a conditional do-not-release guard, and 6/7 gold FLAG_RISK cases scored as SAFE_CAUTION (items 3, 5, 11, 13, 21, 24; item 17 correctly flagged).

So: treat 0% state as “the model does not speak our state schema + often fails to pin settlement,” and treat the six human-inspected cases as the evidence that this can become merchant capital / inventory loss if wired into an agent.

Reproduce (primary path)

pip install -r requirements.txt
python scripts/run_eval.py --model Qwen/Qwen2.5-3B-Instruct

Optional Colab mirror: `notebooks/wa_commerce_blindspot_eval.ipynb` (same protocol; run_eval.py is the source of truth for Hub reproducibility).

Results (this run)

Artifacts:

• —`results/summary.json`
• —`results/evaluation_results.json` / `.csv`
• —`results/human_notes.md` — six striking failure modes

Story in one line: risk looks “ok” because the model defaults to SAFE_CAUTION, but settlement grounding is weak and several high-stakes West African scam patterns are under-flagged.

3. Path forward (speculative)

Data curation

1. 1.Contrastive aspect pairs: minimal edits that only flip don ↔ wan / I go / never, with opposite gold actions; train with DPO so premature “release goods” is preferred against.
2. 2.Lived-experience red-teaming: anonymized dispute chats with Lagos/Accra micro-vendors and agent operators (consent + PII scrub).
3. 3.Rail-aware labels: annotate buyer-debit vs merchant-credit as separate facts, never collapse them.
4. 4.Canonical state vocabulary in the prompt + constrained decoding so automatic metrics measure meaning, not free synonym choice—then re-score semantic errors separately.

Architecture / control

Add a state verification scaffold (or small auxiliary head) that must emit {INITIATED, PENDING_CLEARANCE, CONFIRMED_SETTLED, DISPUTED, REVERSED} before any tool call that releases goods or triggers refunds. Refuse action unless CONFIRMED_SETTLED is supported by receiver-side evidence.

Repository layout

fatima/
  README.md                 # Q1–Q3 answers for the fellowship
  requirements.txt
  data/eval_dataset.json    # 24-item diagnostic set
  scripts/run_eval.py       # PRIMARY reproducible entrypoint
  notebooks/...ipynb        # optional Colab mirror
  results/                  # scored outputs + human_notes

Citation / model card pointer

Evaluation target: Qwen Team — Qwen2.5-3B-Instruct on the Hugging Face Hub. This repo does not re-host model weights; it links the Hub model and hosts the diagnostic set + eval code + run outputs.