Modelpublic

ymoslem/ModernBERT-base-TeleQnA-router-qe-classifier-binary-10ep-lr2e-05-gemma4e4b-5runs_1eval

sourceHugging Faceapache-2.0updated 21d agoView on Hugging Face

0likes32downloads

Model Card

TeleQnA quality estimator for Gemma4-E4B

Binary quality estimator over Gemma4-E4B answers on TeleQnA. Given a question and the model's own answer, it predicts whether that answer is correct (accept) or should be escalated to a stronger model (route).

Companion to the Qwen3-4B-Instruct estimator. A quality estimator reads the efficient model's own output, so it is specific to that model; on a single A100 the Stage 1 routing selects Gemma4-E4B where a two-card configuration selected Qwen3-4B-Instruct.

Results

Evaluated on run 0 of the TeleQnA test split, 1,000 questions.

	precision	recall	F1	support
Route	0.54	0.46	0.50	321
Accept	0.76	0.82	0.79	679
accuracy			0.700	1000
macro avg	0.65	0.64	0.64

Predicting accept for everything would give accuracy 0.679 and macro F1 0.404, so the gain is real but moderate. Route recall of 0.46 means it misses over half the answers that should escalate.

Matched against the Qwen3-4B-Instruct estimator, trained identically:

	Qwen3-4B-Instruct	Gemma4-E4B
accuracy	0.698	0.700
F1 macro	0.635	0.638

Every figure agrees within 0.005, so swapping the efficient tier costs nothing in estimator quality; the ceiling is a property of the task.

Training

cre qe-train from CRE-Router, on ModernBERT-base: 10 epochs, learning rate 2e-5, max length 512, batch 64, early stopping patience 4, class weights on. Trained on all five runs of the train split (45,000 rows) with run 0 of the test split for early stopping, matching the Qwen3-4B-Instruct estimator exactly so the two remain comparable.

Generations came from gemma-4-E4B-it under vLLM 0.19.0 on 1 x A100 80GB at concurrency 32.