esmaeil-abedi-dev/verifier-ood-qwen2.5-0.5b

verifier-ood-qwen2.5-0.5b

The out-of-distribution (domain hold-out) variant of the verifier-CE authorization model. Same method and base model as `verifier-ce-qwen2.5-0.5b`, but deliberately trained on only 3 of the 5 domains (email, payment, repo) with file and db held out, so its transfer to unseen domains can be measured honestly.

Companion to the paper "Learnable Authorization: A Verifier-as-Reward Benchmark and Method for AI-Agent Authority." Code and full experiment log: https://github.com/esmaeil-abedi-dev/verifier-as-reward

Why this model exists

An out-of-distribution test is only valid if the test domains were never seen in training. The main released model saw all 5 domains, so it cannot be used to measure domain transfer. This model is trained without file/db precisely so it can be evaluated on file/db as genuinely unseen domains.

Evaluation (this checkpoint, seed 8)

Train→OOD gap ≈ 1.5 points (across 3 seeds: held-out 0.968 ± 0.001, train 0.983). The model transfers to entirely unseen action namespaces and resource formats at near-parity — evidence that verifier-CE learns domain-invariant authorization structure (delegation, attenuation, revocation, expiry, budgets) rather than per-domain surface patterns. Residual error concentrates in the two hardest structural classes (chain_structure, scope_escalation).

Intended use and limitations

• —Research artifact for the domain-transfer experiment — the main all-domains model is the one to use for general inference.
• —Synthetic benchmark, single verifier as ground truth; transfer is shown across domain vocabulary within the attenuated-delegation formalism, not across different authorization formalisms.
• —See the repository's EXPERIMENT_LOG.md for the full OOD result and the two-models rationale (§1b).

Usage

PYTHONPATH=. python train_verifier_reward.py \
    --eval-checkpoint esmaeil-abedi-dev/verifier-ood-qwen2.5-0.5b \
    --test-file benchmark_test.jsonl