Modelpublic

Baragi-AI/Munche-768-AI-Detector

sourceHugging Facegemmaupdated 2mo agoView on Hugging Face

3likes39downloads

Model Card

⚠️ Do not use this model as evidence of AI authorship.

Munche-768-AI-Detector was built for dataset triage and proof-of-concept research. It cannot establish plagiarism, copyright infringement, misconduct, or whether a person or an AI wrote a text. Its output can be wrong.

Munche-768-AI-Detector

Munche-768-AI-Detector classifies Korean genre-fiction passages as human, uncertain, or llm. It starts from Munche-768, then jointly tunes LoRA weights in the top four Transformer layers and a 769-parameter linear classifier.

Demo

**Run Munche-768-AI-Detector in your browser**

Overall test result

The sealed test combines 277 human passages and 256 LLM passages from the independent-generation and content-preserving rewrite evaluations.

Metric	Result
AUROC	98.59%
Binary accuracy	94.00%
Balanced accuracy	93.91%
Human recall	96.03%
LLM recall	91.80%
Human false-positive rate	3.97%

Independent-generation test

The sealed test contains 87 passages from human-written novels and 66 passages written directly by 11 language-model families. Content-preserving rewrites are excluded from this evaluation.

Metric	Result
Binary accuracy	100.00%
Balanced accuracy	100.00%
Human recall	100.00%
LLM recall	100.00%
Human false-positive rate	0.00%

Three-way decision

The thresholds were selected on validation data only. The classification score is not a calibrated probability that a passage was written by AI. Results below cover the full sealed test.

Score	Output
`≤ 0.5000`	Human
`0.5000 < score < 0.8697`	Uncertain
`≥ 0.8697`	LLM

Metric	Result
Coverage	93.62%
Accuracy among classified passages	95.19%
Human classified as LLM	1.08%
Human classified as uncertain	2.89%
LLM classified as human	8.20%
LLM classified as uncertain	10.16%

Within the independent-generation subset, all 87 human passages received a human decision. Of the 66 LLM passages, 64 received an LLM decision and two were uncertain.

Content-preserving rewrite test

This test contains 190 human passages and 190 LLM rewrites that preserve the source content. It is harder than distinguishing independently written human and LLM fiction.

Metric	Binary decision	Three-way decision
AUROC	97.47%	N/A
Balanced accuracy	91.58%	N/A
Human recall	94.21%	N/A
LLM recall	88.95%	N/A
Human classified as LLM	5.79%	1.58%
LLM classified as human	11.05%	11.05%
Coverage	N/A	91.58%
Accuracy among classified passages	N/A	93.10%

Input length

Use passages between 384 and 2,048 EmbeddingGemma tokens. Inputs shorter than 384 tokens are not supported as stable operating inputs. Inputs longer than 2,048 tokens must be divided into separate windows before classification.

The training and evaluation corpora covered short rewrite passages near 400 tokens and independent fiction passages near the 2,048-token model limit. Document-level aggregation across multiple windows has not been calibrated.

Usage

Access to the gated EmbeddingGemma base model is required.

bash

pip install torch numpy sentence-transformers peft safetensors

python

from inference import MuncheAIDetector

detector = MuncheAIDetector(".")
result = detector.predict(korean_fiction_passage)

print(result)
# {"label": "human" | "uncertain" | "llm", "score": float, "tokens": int}

Training data

Split	Human	LLM
Train	1,315	1,135
Validation	290	275
Test	277	256

The training set combines human-written Korean genre fiction, independently generated LLM fiction, and content-preserving LLM rewrites. GPT-5.6 Sol Medium contributes 24 training passages and six validation passages. No Sol Medium passage was added to test.

The detector was initialized from Munche-768. Only LoRA weights in Transformer layers 20-23 and the linear classifier were updated. The selected checkpoint is step 275. A preservation loss limited movement away from the original Munche-768 embedding during tuning.

Raw human fiction is not distributed with this repository.

Limitations

—The model was trained and evaluated on Korean genre fiction.
—Generalization to language-model families absent from training remains unknown.
—The model cannot identify text jointly written or substantially edited by humans and AI.

License

Munche-768-AI-Detector is derived from google/embeddinggemma-300m and Munche-768. Use is subject to the Gemma license and the access terms of the gated base model.