CoolFace
Apppublic

hugging-apps/open-jev-deberta-v3-large-demo

sourceHugging Faceupdated 4d agoView on Hugging Face
0likes
App README

open-jev — typed decisions in one forward pass

An interactive demo for `com-kotobalabs/open-jev-deberta-v3-large`, an open, Jev-shaped decision model: a DeBERTa-v3-large encoder (≈434M params) with a span-matching scoring head that reads a piece of state text and answers any number of typed questions about it — all in a single forward pass.

Three question kinds:

kindmeaningread-out
choiceone of N unordered options (2–255)the argmax option + the full distribution + confidence
scoreone of 2–10 ordered levelsthe expected level (a real number, may fall between levels) + distribution + confidence
noula yes/no statement (fixed no / yes options)p(yes)

Nothing is generated, so there is no structured-output error to make: the model only ranks the options you hand it and cannot write prose.

What to expect

Published measurements on the authors' test sets are 0.854 accuracy in-domain and 0.690 out-of-distribution (new instructions, new option sets), with ECE 0.022 / 0.035. Treat a distribution as a ranked shortlist rather than ground truth. The model reads the question only partly and is weakest on ordered scales it has not seen before. English only, 512-token context (the first 256 tokens of the state are read); the three training domains are banking support, movie reviews and Wikipedia yes/no — anything else is out of distribution and should be measured before you rely on it.

Implementation notes

  • —Inference uses the authors' own typed_decisions package, vendored verbatim from the model repo (identical to `kotoba-lang/typed-decisions`, Apache-2.0), so pip install typed-decisions is not required.
  • —Weights load once at startup and stay resident on the GPU; each request is one forward pass of a 24-layer / 1024-hidden encoder, so it is milliseconds-to-seconds of compute, but the first call after an idle period also streams the weights into VRAM on the shared ZeroGPU worker.
  • —The temperature slider defaults to 1.05, the post-hoc value the authors fitted on their validation split. It changes how peaked the probabilities are, never the ranking.

Example data

The gr.Examples rows are taken verbatim from the model authors' own Apache-2.0 test sets (data-fam/test.jsonl and data-fam/ood-test.jsonl in `kotoba-lang/typed-decisions`), so the inputs are exactly the ones the published measurements were taken on. SST-5 rows had their raw token spacing normalised for readability; predictions were verified to be unchanged.

Credits

Model and code: kotoba-lang (Apache-2.0), built on `microsoft/deberta-v3-large`.