Apppublic

di35117/msme-linguistic-decoding-env

sourceHugging Faceupdated 5mo agoView on Hugging Face

0likes

App README

🧠 Linguistic Decoding RL

A reinforcement learning environment for long-horizon hidden state inference — teaching an LLM agent to decode what entities actually mean, not just what they say.

The Core Problem

Language models are remarkably good at face value. Ask them what a message means and they will tell you what the words say. Ask them what the speaker intends, what they are concealing, or how their story has shifted over three conversations — and they fail. They anchor to the surface.

This is a fundamental gap. In the real world, the most important information is rarely stated directly:

—A borrower who says "collections have been a bit slow but we're managing" may be three months from default.
—A founder who says "we're being strategic about burn" may have six weeks of runway.
—The truth lives not in the words but in the pattern across time — the drift in tone, the gaps in documentation, the meetings that keep getting cancelled.

Linguistic decoding is the skill of inferring latent intent, hidden state, and real motive from what is said, how it is said, what is avoided, and how all of that changes across a temporal sequence. Teaching this to an AI system is a vast, open research problem. There is no clean dataset, no obvious loss function, and no single domain where it is fully solved.

Scope — What We Built for This Hackathon

Linguistic decoding as a general problem is enormous. Narrowing it to something tractable within a hackathon timeline — while still producing real, meaningful results — required a deliberate choice of domain.

We chose Indian MSMEs and startups for a specific reason: both are high-stakes, information-asymmetric contexts where the cost of misreading intent is concrete and measurable. A lender or investor who takes language at face value in these settings does not just make a bad prediction — they make a bad financial decision with real consequences. The problem is also structurally interesting because the two populations have opposite communication biases: MSME borrowers in India systematically understate distress (cultural and social pressure to appear in control), while startup founders systematically overstate health (investor signalling incentives). An agent that can navigate both — learning to condition its interpretation on who is speaking before deciding what they mean — is doing something genuinely non-trivial.

This is the slice we modelled, trained on, and evaluated. The environment, the world generator, the message templates, the behavioral profiles, and the reward structure are all calibrated to this specific context. The results are real within this scope. Generalizing the framework to other domains — legal negotiation, medical disclosure, conflict resolution, political communication — is the natural next step, but that is future work.

Why RL for This

Supervised learning teaches an agent to match outputs to labels. It cannot teach an agent to reason under uncertainty across time — to hold a hypothesis about a speaker's true state, update it as new signals arrive, and commit to an action whose consequences only materialize steps later.

This is a long-horizon planning problem. The right intervention at step 30 depends on what happened at steps 1 through 29. A premature escalation on a healthy entity is as damaging as inaction on a failing one. The agent must learn not just what to infer but when to act on it — and how much weight to give a single message versus an accumulated pattern.

GRPO-based training over multi-step episodes with delayed episode rewards makes this tractable — rewarding the agent for the quality of its full trajectory, not just individual decisions.

What the Agent Learns

At each step, the agent:

1.Receives a biased natural language message from the entity
2.Observes behavioral proxies — response latency, document completion rate, meeting cancellations, escalation avoidance
3.Maintains a running estimate of the hidden stress level across the episode: healthy → watch → substandard → doubtful → loss
4.Selects a policy action from the intervention menu
5.Receives a step reward for action appropriateness and an inference bonus for correctly identifying the hidden state
6.Refines its policy across episodes via GRPO

The ground truth is never revealed during the episode. The agent must earn its estimate from accumulated evidence — exactly as a real analyst would.

World Modelling

The environment generates a fully synthetic but internally consistent world for each episode. Every episode is a new entity — new hidden state, new financial profile, new behavioral disposition, new speaker bias. The world is not static; it evolves as the agent acts on it.

Each entity is characterized by:

—A true stress level drawn from a calibrated distribution per domain — invisible to the agent
—A financial snapshot — revenue growth, cash runway, debt service coverage, receivables overdue, burn rate — consistent with the stress level but not directly observable
—A behavioral profile — response latency, document completion, meeting cancellations, escalation avoidance — which drifts as the entity's condition changes across steps
—A speaker bias — the systematic tendency to understate (MSME) or overstate (startup) true condition, injected into every generated message

The world evolves across steps. A well-timed field visit slows deterioration. Inaction accelerates it. Messages and behavioral signals update each step to reflect the new underlying state. The agent is not reading a static transcript — it is navigating a living system that responds to its choices.

The world modelling is what prevents the agent from memorizing patterns. It must generalize a reasoning process that works across varied entities, bias levels, financial conditions, and deterioration trajectories.

Environment Design

Domains

Domain	Speaker Bias	Sectors
MSME	Understatement	retail, manufacturing, agri-processing, logistics, hospitality
Startup	Overstatement	fintech, edtech, healthtech, SaaS, consumer

Training across both domains is intentional — opposite bias directions force the agent to condition on who is speaking before interpreting what they say.

Stress Levels (Hidden State)

healthy → watch → substandard → doubtful → loss

Intervention Actions

| Action | Best Used When | @@ -109,10 +101,10 @@ Each level maps to calibrated financial snapshots, behavioral profiles, and mess

Reward Structure

—Step reward — action appropriateness against true stress level (-1.0 → +1.0)
—Inference bonus — +0.4 for correctly naming the hidden stress level; partial credit for adjacent estimates
—Critical miss penalty — -0.5 for do_nothing or schedule_follow_up on doubtful or loss entities
—Episode reward — trajectory bonus for net stress reduction; terminal penalty for entities that end worse than they started

@@ -133,8 +125,8 @@ Domain Adapter Registry domains/_init.py ▼ MSME + Startup Adapter domains/msmestartup/adapter.py │ ├──▶ worldgenerator.py — hidden state + financial + behavioral synthesis ├──▶ messagegenerator.py — biased NL message generation per step ├──▶ reward.py — step + episode reward logic ├──▶ network.py — peer entity contagion effects └──▶ memory.py — cross-step state accumulation @@ -144,9 +136,7 @@ MSME + Startup Adapter domains/msme_startup/adapter.py

Training Results

We trained for 30 episodes with each episode capped at 90 steps, fine-tuning a Qwen 1.5B model with GRPO on this reinforcement learning task. Both numbers reflect resource constraints, not design ceilings — the environment is built for trajectories of 300–500 steps where the long-horizon dynamics are fully expressed. Within these limits, the goal was proof of concept: does the reward signal improve, does the policy stabilize, does the agent develop domain-differentiated strategies across MSME and startup profiles.

The answer to all three is yes.

Reward climbs through the first 15 episodes driven primarily by the agent unlearning high-penalty defaults — particularly do_nothing when behavioral signals are deteriorating. KL divergence stays bounded throughout, indicating stable updates without policy collapse.

Baseline vs. Trained Distribution

[image]

The untrained baseline defaults to low-commitment actions regardless of signal. After training, the reward distribution shifts toward +0.3 → +0.8. The negative tail shrinks but persists — heavily biased loss-level entities remain the hardest case, as expected.

Training Metrics Dashboard

@@ -181,33 +171,24 @@ This view isolates the reward trend with a moving average, making it easier to s

Training & Evaluation Workflow

bash

# Baseline
py -3 scripts/run_baseline_eval.py --episodes 30 --output artifacts/baseline_rewards.json

# Train
py -3 train_grpo.py --episodes 30 --max_steps 90 --output_dir msme_rl_checkpoints




# Judge artifacts
py -3 scripts/generate_judge_artifacts.py \
    --training_json msme_rl_checkpoints/reward_curve.json \
    --baseline_json artifacts/baseline_rewards.json \
    --output_dir artifacts




# Deterministic eval

py -3 scripts/run_deterministic_eval.py --seed 123 --episodes 5 \
    --output artifacts/deterministic_eval.json

# Pre-submission check
py -3 scripts/pre_submit_check.py

@@ -221,19 +202,19 @@ msmeEnv/ ├── openenv.yaml ├── pyproject.toml ├── _init.py ├── traingrpo.py ├── worldgenerator.py ├── reward.py ├── network.py ├── memory.py ├── messagegenerator.py ├── server/ │ ├── app.py │ └── msmeEnvenvironment.py ├── domains/ │ ├── init.py │ └── msmestartup/ │ └── adapter.py └── scripts/ ├── runbaselineeval.py ├── eval.py @@ -244,20 +225,14 @@ msmeEnv/

What Comes Next

The 90-step cap was a resource limitation, not a ceiling. At 300–500 steps, the long-horizon dynamics become fully visible — entities that appear healthy at step 20 but deteriorate by step 80, requiring the agent to hold and update its hypothesis across a much longer evidence window. That is where the real test of linguistic decoding lives.

Beyond longer trajectories, the framework extends naturally to other high-stakes, information-asymmetric domains — legal negotiation, medical disclosure, HR conflict, political communication — anywhere that what is said and what is meant systematically diverge. The Indian MSME and startup context was the right scope for a hackathon. The architecture is built to go further.