Mitchins/fiction-nonfiction-deberta-v3-xsmall-v2

V2.19 fiction / nonfiction classifier

Model summary

Approved V2.19 release model for binary fiction/nonfiction classification.

• —promoted model: DeBERTa-v3-xsmall V2.19 release candidate
• —task: binary text classification (fiction vs nonfiction)
• —threshold mode: temperature_scaled
• —fitted temperature on validation logits: 2.1562
• —selected threshold: 0.35
• —threshold source: guarded calibrated validation sweep within 0.005 macro-F1 of the best threshold

This release also includes a fallback model:

• —fallback model: TinyBERT 6L V2.19 release candidate
• —threshold mode: guarded_raw
• —selected threshold: 0.55

Intended use

Recommended inference granularity is paragraph/chunk/window classification rather than whole-book single-pass inference.

Use this model to classify 512-token text windows into fiction or nonfiction.

Operational definition:

• —fiction = actual story prose only, including excerpts and narrative scenes
• —nonfiction = metatext, glossary/reference material, legal/copyright pages, author bios, reviews, explanatory text, and other non-story prose
• —exclude = junk, mixed, or uncertain material used only in review/auditing, not as a runtime label

Not intended for:

• —factuality detection
• —authorship attribution
• —literary quality assessment
• —truthfulness classification

Anti-leakage protocol

The evaluation protocol includes:

• —canonical-work split isolation
• —near-duplicate filtering
• —Jaccard similarity leakage audits
• —hard negative mining
• —disagreement-based relabel review

Training data / evaluation summary

Training used the final V2.19 cleaned dataset after:

• —approved patch manifest application
• —two finite disagreement-review smoothing rounds

Applied dataset changes:

• —53 relabels
• —1 exclusion

Evaluation summary for the promoted model:

• —test macro-F1: 0.9338
• —OOD macro-F1: 0.9327
• —fresh stress macro-F1: 0.9718
• —validated metatext recall: 1.0000 on the maintained 21-row V2.19 acceptance slice
• —structural nonfiction recall: 0.9952
• —fresh story-prose false-nonfiction rate: 0.0316
• —nonfiction precision: 0.9542

The fallback TinyBERT 6L model also passed release gates, with:

• —test macro-F1: 0.9339
• —OOD macro-F1: 0.9332
• —fresh stress macro-F1: 0.9533
• —validated metatext recall: 0.9000 on the maintained V2.19 acceptance slice
• —structural nonfiction recall: 0.9857
• —fresh story-prose false-nonfiction rate: 0.0484
• —nonfiction precision: 0.9604

Thresholds and calibration note

The promoted model was calibrated with temperature scaling and uses temperature_scaled operating mode. Calibration was acceptable but not perfect:

• —test ECE: 0.0166
• —OOD ECE: 0.0196
• —fresh-stress ECE: 0.0425
• —fresh-stress Brier: 0.0222

Recommended Operating Points

The model was calibrated using temperature scaling (temperature = 2.1561811131629827). Different thresholds trade off:

• —fiction protection
• —nonfiction recall
• —overall macro-F1

Recommended default:

• —use 0.35 after temperature scaling unless your application strongly prioritizes minimizing story-prose false nonfiction errors.

Validation best-threshold reference:

• —best validation macro-F1: 0.9384 at threshold 0.10
• —selected validation macro-F1: 0.9335 at threshold 0.35

Do not assume the default 0.5 threshold reproduces the reported operating point.

Limitations

• —This is a binary classifier only.
• —The validated metatext slice is small (21 rows), so acceptance confidence is narrower than on the main test/OOD splits.
• —The structural nonfiction acceptance slice is also small (422 rows).
• —Nonfiction precision is above threshold but with a limited margin (0.9542 vs 0.9500).
• —Fresh story-prose acceptance still contains some contamination/non-story material.
• —Performance is reported for the V2.19 release distribution and should not be treated as a guarantee on unrelated domains.

Fallback path

If the promoted model is unavailable, use:

• —TinyBERT 6L V2.19 release candidate

That fallback is the released TinyBERT 6L candidate and remains the closer substitute within this repo.

Supporting artifacts

Supporting evaluation artifacts are included in the release bundle.

Risk notes

• —No unresolved medium/high leakage concern was found in the release audit.
• —Final dataset build reported canonical overlaps 0, rows with Jaccard >= 0.80 0, and max train/eval Jaccard 0.5200.
• —The model is intended for release-consistent use on similar text classification inputs, not as a general literary or factuality detector.

License

Released for research/evaluation use only. See LICENSE.