igorls/gemma4-e4b-classifier

1---2license: gemma3base_model: google/gemma-4-E4B-it4tags:5- gemma6- gemma-47- classification8- text-only9- vram-optimized10- ollama11language:12- en13- multilingual14library_name: transformers15pipeline_tag: text-generation16---17 18# Gemma 4 E4B Classifier (vision/audio-stripped)19 20A modality-stripped variant of [`google/gemma-4-E4B-it`](https://huggingface.co/google/gemma-4-E4B-it) for **text-only classification, entity extraction, and structured-memory extraction**. The vision encoder (~150M params) and audio encoder (~300M params) are removed; the text path is unchanged.21 22**Headline:** Same instruction-tuned text behavior as the official Gemma 4 E4B-it — including its multilingual coverage — but at **6.5 GB resident VRAM instead of 10.6 GB** (Ollama Q4_K_M, RTX 3090, Linux). All safety alignment is preserved — this is **not** an abliterated or uncensored variant.23 24Fits comfortably on **8 GB GPUs at Q4_K_M** with realistic context lengths (5.85 GB resident at ctx=4096, 5.96 GB at ctx=8192). The official multimodal Q4_K_M sits at 10.2 GB resident even at ctx=8192 and won't load on 8 GB cards.25 26## Why this exists27 28Gemma 4 E4B is the local leader on small-model classification tasks (room classification, entity/memory extraction). It locks out users with 12 GB GPUs because the official Q4_K_M is 10.6 GB resident — the vision + audio encoders sit in VRAM whether you use them or not. For text-only workloads, those modality encoders are dead weight.29 30This variant strips them via clean re-instantiation: load the multimodal checkpoint, copy text-path tensors into a fresh `Gemma4ForCausalLM(text_config)`, save. No safety-alignment changes. No retraining. No surgery on safetensors files.31 32## How it compares33 34Measured on RTX 3090, Ollama 0.x, against the MemPalace small-model benchmark harness (n=100 per task):35 36| Task | Official `gemma4:e4b-it-q4_K_M` | This model (Q4_K_M) | Δ |37|---|---:|---:|---:|38| Calibration | 1.0000 | **1.0000** | 0.0000 |39| Room classification (closed-set) | 0.6200 | **0.6200** | 0.0000 (exact tie) |40| Room classification (open-set) | 0.6556 | 0.6526 | -0.0030 |41| Entity extraction (F1) | 0.7519 | 0.7318 | -0.0201 |42| Memory coverage | 0.9125 | **0.9375** | +0.0250 (higher) |43| **VRAM resident** | **10626 MB** | **6517 MB** | **-4109 MB** |44| e2e p50 (closed-set room) | 230.9 ms | 232.4 ms | +1.5 ms (noise) |45 46All accuracy deltas are within statistical noise at n=100. The 4.1 GB VRAM win is real and reproducible.47 48## Multilingual robustness49 50The strip preserves the base model's multilingual capability. Same classification + extraction tasks were run with inputs translated into Portuguese (pt-BR), Spanish (es), and Chinese (zh) — labels and the slug taxonomy kept in English to test the realistic cross-lingual mapping case. Scoring uses `embeddinggemma` for semantic similarity so cross-lingual cosine isn't artificially penalized.51 52| Task | en | pt-BR | es | zh |53|---|---:|---:|---:|---:|54| Calibration | 1.000 | 0.950 | 0.950 | 0.950 |55| Room classification (closed-set) | 0.624 | 0.584 | 0.584 | 0.584 |56| Room classification (open-set) | 0.676 | 0.636 | 0.641 | 0.639 |57| Entity extraction (F1) | 0.732 | 0.747 | 0.747 | 0.694 |58| Memory coverage | 0.912 | 0.850 | 0.850 | 0.912 |59 60Closed/open room classification stays within ±0.02 across all four languages; entity F1 within ±0.05; memory coverage within ±0.06. The strip did not introduce a multilingual regression. Models still emit responses in the input language by default — if your application needs same-language extraction (e.g. memories phrased in Portuguese for Portuguese conversations), the model does that natively.61 62## What was actually dropped63 64From the 7996.2M-parameter multimodal checkpoint:65 66| Module | Params dropped |67|---|---:|68| `model.audio_tower.*` (USM-style conformer) | 304.8M |69| `model.vision_tower.*` (MobileNet-v5 lineage) | 167.4M |70| `model.embed_audio.*` (audio→text soft-token projector) | 3.9M |71| `model.embed_vision.*` (vision→text soft-token projector) | 2.0M |72| **Total dropped** | **478.1M (6.0%)** |73| **Total kept** (text path) | **7518.1M (94.0%)** |74 75The VRAM saving (4.1 GB) is significantly larger than the dropped weights account for (~250 MB at Q4_K_M). The remainder comes from: modality encoders kept at higher precision than Q4 inside the GGUF, activation buffers sized for image-token sequences (up to 1120 tokens/image), and the multimodal embedders' vocab-offset tables.76 77## Quantization variants78 79- **`Q4_K_M`** (5.3 GB on disk, 6517 MB resident) — recommended default.80- **`Q8_0`** (8.0 GB on disk) — precision comparator; minimal accuracy lift on classification.81- Source safetensors (this repo at bf16, 13.92 GB).82 83## Usage84 85### Hugging Face Transformers86 87```python88from transformers import AutoTokenizer, Gemma4ForCausalLM89import torch90 91tok = AutoTokenizer.from_pretrained("igorls/gemma4-e4b-classifier")92model = Gemma4ForCausalLM.from_pretrained(93    "igorls/gemma4-e4b-classifier",94    torch_dtype=torch.bfloat16,95    device_map="cuda",96)97 98messages = [{"role": "user", "content": "What is the capital of France? One word."}]99chat = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)100ids = tok(chat, return_tensors="pt").input_ids.to("cuda")101out = model.generate(ids, max_new_tokens=10, do_sample=False)102print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))103```104 105### Ollama106 107```bash108ollama pull igorls/gemma4-e4b-classifier:Q4_K_M109ollama run igorls/gemma4-e4b-classifier:Q4_K_M "What is the capital of France?"110```111 112For classification workloads, pass `"think": false` at the top level of the `/api/generate` request to disable Gemma 4's CoT mode (which otherwise consumes the `num_predict` budget):113 114```bash115curl http://localhost:11434/api/generate -d '{116  "model": "igorls/gemma4-e4b-classifier:Q4_K_M",117  "prompt": "Classify into one word (indoor, outdoor): The kids are playing in the backyard.",118  "think": false,119  "stream": false,120  "options": {"temperature": 0, "num_predict": 16}121}'122```123 124## Safety surface125 126This variant is **safety-aligned identically to the official `gemma-4-E4B-it`**. The strip does not touch the text-path weights where alignment lives; it only removes the unused modality encoders.127 128Validated on 18 raw NSFW classification samples (closed-set room, open-set slug invention, entity extraction with named entities, structured memory extraction with decisions/preferences/facts/commitments):129 130- **Zero refusals** on any sample.131- **JSON validity 100%** on the structured extraction tasks.132- **Open-set slugs are functional** rather than euphemistic.133 134This confirms the architectural insight from prior research: safety alignment doesn't surface on classification surfaces regardless. There's no reason to ship an uncensored variant for these workloads.135 136## Limitations137 138- **Text-only.** No vision input. No audio input. The encoders are gone. Passing image or audio tokens will produce undefined behavior.139- **Same context window as base** (128k tokens).140- **Same tokenizer.** The vocab includes vision/audio special tokens (`<image>`, `<audio>`, etc.) for compatibility with the official tokenizer; these tokens won't activate any modality processing in this variant.141- **No MTP drafter support on Ollama yet.** Upstream llama.cpp doesn't recognize the `Gemma4AssistantForCausalLM` architecture as of May 2026, so Ollama on Linux/CUDA can't pair this target with the official MTP drafter. For MTP-accelerated inference, use Transformers or vLLM directly — see the [MTP acceleration](#mtp-acceleration) section below.142 143## MTP acceleration144 145The official MTP drafter [`google/gemma-4-E4B-it-assistant`](https://huggingface.co/google/gemma-4-E4B-it-assistant) (78M params, activation-aware) pairs cleanly with this stripped target. Output is lossless (byte-identical at deterministic decode). Measured on RTX 3090 via HF Transformers:146 147| Prompt shape | Tokens generated | Baseline | + MTP drafter | Speedup |148|---|---:|---:|---:|---:|149| MCQ single letter | 5 | 394 ms | 363 ms | 1.09x |150| Open Q one-word | 5 | 395 ms | 249 ms | 1.59x |151| Slug classification | 5 | 462 ms | 224 ms | 2.07x |152| JSON entity list (128 tok) | 128 | 12291 ms | 6712 ms | 1.83x |153| JSON memories (114 tok) | 114 | 8425 ms | **2771 ms** | **3.04x** |154 155Speedup tracks output predictability — structured JSON outputs land at the high end (3x), short slug/letter classifications around 1.5-2x, free-form continuations near 1x.156 157```python158from transformers import AutoModelForCausalLM, AutoTokenizer159import torch160 161target = AutoModelForCausalLM.from_pretrained(162    "igorls/gemma4-e4b-classifier",163    dtype=torch.bfloat16,164    device_map="cuda",165)166drafter = AutoModelForCausalLM.from_pretrained(167    "google/gemma-4-E4B-it-assistant",168    dtype=torch.bfloat16,169    device_map="cuda",170)171tok = AutoTokenizer.from_pretrained("igorls/gemma4-e4b-classifier")172 173messages = [{"role": "user", "content": "Classify into one word (indoor, outdoor): The kids are playing in the backyard."}]174chat = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)175ids = tok(chat, return_tensors="pt").input_ids.to("cuda")176 177out = target.generate(178    ids,179    assistant_model=drafter,180    max_new_tokens=20,181    do_sample=False,182)183print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))184```185 186For a self-hosted OpenAI-compatible HTTP endpoint, wrap the pair in a small FastAPI server that holds both models resident and exposes `/v1/chat/completions`. Example: [`scripts/08_mtp_server.py`](scripts/08_mtp_server.py) in the source repo, callable as:187 188```bash189curl http://localhost:8765/v1/chat/completions -d '{190  "model": "igorls/gemma4-e4b-classifier",191  "messages": [{"role":"user","content":"What is the capital of France?"}],192  "max_tokens": 16,193  "use_mtp": true194}'195```196 197### vLLM (future)198 199vLLM is the right inference stack for production throughput — it implements the drafter's centroid-masking optimization (sparse lm_head over ~4K candidates instead of ~262K vocab, ~45x reduction in lm_head compute):200 201```bash202vllm serve igorls/gemma4-e4b-classifier \203  --speculative-config '{"model": "google/gemma-4-E4B-it-assistant", "num_speculative_tokens": 4}'204```205 206**However**, as of May 2026 (vLLM 0.20.2, latest on PyPI), this fails: the drafter's `Gemma4AssistantConfig` is not yet registered in vLLM's `AutoModel` mapping. The vLLM Gemma 4 recipes page documents the feature but it's ahead of the released version. Track [vllm-project/vllm](https://github.com/vllm-project/vllm/) for the release that lands `Gemma4Assistant` support; once available, the command above should work as-is against this model.207 208## License209 210Inherited from the base model: [Gemma Terms of Use](https://ai.google.dev/gemma/terms). By using this model you agree to those terms.211 212## Citation213 214This is a derivative work of Google's Gemma 4 E4B. If you use it, please also credit:215 216```217@misc{gemma_2025,218  title={Gemma 4 Technical Report},219  author={Google DeepMind},220  year={2026},221  url={https://huggingface.co/google/gemma-4-E4B-it},222}223```224 225## Acknowledgments226 227- **Google DeepMind** for Gemma 4 and the open-weight release.228- The **MemPalace small-model benchmark research** (PR #1447) that surfaced the VRAM gap and motivated this work.229- The **`igorls/gemma-4-E4B-it-heretic-GGUF`** (author's prior abliteration experiment) for accidentally demonstrating the architectural VRAM win that this artifact reproduces through a clean, safety-aligned path.230