mzio/aprm-thought-generations-snorkel-finance

aprm-thought-generations-snorkel-finance

Act-PRM inferred-thought generations for snorkel_finance agent traces (Qwen3-4B-Instruct-2507). For each logged (state s, action x), an offline EM samples G=4 candidate thoughts z, scores each by the length-penalized action likelihood, and commits the top-1.

generations/{policy,base,policy_last,base_last}.jsonl (primary)

One row per logged action step. Full candidate pool so you can take top-1 OR recompute any weighting:

• —thoughts (G): candidate thoughts z
• —likelihoods (G): p(x | s, z), policy per-action-token likelihood
• —rewards (G): length-penalized score = p(x|s,z) - 0.15 * (|z| / 96)
• —advantages (G): group-normalized EM weights
• —thought_tokens (G): |z| per candidate
• —best: index of the committed top-1 (argmax reward)
• —target_action, sample_id, timestep, split

Variants = EM scorer (policy = LoRA policy likelihood, base = frozen base model) × EM snapshot (_last = final steplast checkpoint; else stepbest).

sft_corpus_top1/<variant>/{train,eval,meta}.json (convenience)

The baked top-1 SFT corpus: assistant target = thoughts[best] + "\n\n" + action.

Result

On next-action prediction (action-subspan ppl), all Act-PRM variants beat both the expert-reasoning oracle and the action-only baseline in hide-obs and full-context. See the act-prm-blog cc-finance-1.x notes/notebooks.