modilify/Modilify-Mk1
121
1---2license: other3license_name: modilify-open-model-license-1.04license_link: LICENSE5library_name: transformers6pipeline_tag: image-text-to-text7tags:8 - diffusion9 - multimodal10 - image-text-to-text11 - mixture-of-experts12 - trust-remote-code13---14 1516 17# Modilify Mk118 19A 26B-A5B multimodal block-diffusion model that thinks in latent space, commits only when it is ready, and was trained on a single Apple silicon machine in less than a day.20 21Modilify Mk1 is not another long-context decoder with a bigger reasoning budget. It is a Transformer-in-Transformer: the heavy DiffusionGemma trunk still sees text, images, and video, while a recurrent latent deliberation stack compresses the entire chain of thought into a compact hidden trajectory. Visible tokens are no longer the only place intelligence can live. The exclusive excess-entropy commit formula decides, every denoise, how many tokens the model is allowed to lock in. Easy problems finish fast. Hard problems keep deliberating.22 23This official release is the first public Mk1 checkpoint. It is materially more stable than [Modilify Mk1 Preview](https://huggingface.co/modilify/Modilify-Mk1-preview), restores the full vision tower, and ships default inference settings that run about **6× faster** than autoregressive models.24 25## Breakthroughs26 27| | |28| --- | --- |29| **One Mac. One day.** | Trained on a **single Apple silicon** machine in **less than 24 hours**. |30| **Seven million tokens.** | The adaptation used about **7 million training tokens**, not a web-scale second pretrain. |31| **Intelligence density** | Far more capability per activated parameter, and far more capability per training token, than a conventional post-train at this size. |32| **Transformer-in-Transformer** | A latent Transformer sits inside every heavy denoise and writes a recurrent memory that survives canvas commits. |33| **Latent CoT compression** | Chain-of-thought is compressed into token latents and 64 memory slots instead of being dumped into visible tokens. |34| **Exclusive commit formula** | Excess-entropy fusion, `p²` when entropy is honest, a hard prefix-risk budget, and a stagnation jump. Not a confidence threshold. |35| **6× default throughput** | Default settings target speed. Preview's quality-oriented knobs are still available when you want them. |36| **Adjustable inference speed** | Move `commit_failure_budget`, `denoise_temperature`, and the ponder / jump limits. Same weights, different operating point. |37| **More stable than Preview** | Cleaner commit geometry, stronger latent addressing, and no leftover adapter surface. |38| **Better agents** | Native thinking-channel control, tool-ready Gemma turns, and a latent scratchpad that does not pollute the user-visible transcript. |39 40## Why this is different41 42Most reasoning models buy intelligence with more visible tokens. That is expensive, leaky, and hard to stop. Mk1 buys it with **latent deliberation**:43 441. Each heavy denoise still runs the 26B-A4B MoE trunk over a 256-token rolling canvas.452. A 4-layer latent Transformer reads the noisy canvas, confidence, entropy, and age, then updates per-token latents plus a 64-slot persistent memory.463. That compact state is mapped back through the frozen self-conditioning bridge and conditions the next heavy pass.474. The exclusive commit formula then locks a variable-length prefix. The memory slots do not shift. The thought continues even after the visible tokens have moved on.48 49The result is elastic inference. You can spend more heavy-denoise work on a hard agent turn, or commit more tokens per pass and finish sooner when the problem is easy. Throughput is a configuration choice, not a second model.50 51## Efficiency52 53Mk1 is an argument about **intelligence per parameter** and **intelligence per training token**.54 55The released model activates 4.159B text parameters on a heavy denoise, plus the 570M vision encoder when images or video are present. The latent stack is small. The adaptation that produced this checkpoint ran on **one Apple silicon machine**, finished in **under 24 hours**, and saw about **7 million tokens**. That is not a claim that data does not matter. It is a claim that a better architecture can extract more from each token and each watt.56 57Default inference is the fast operating point. Compared with the slower Preview evaluation settings (`denoise_temperature=0.4`, `commit_failure_budget=0.05`, `jump_on_no_progress_after=32`), the Mk1 defaults are built for about **6× higher throughput**. Tighten the budget if you want Preview-like caution. Loosen it if you want the model to finish.58 59## Model Summary60 61| | |62| --- | ---: |63| Architecture | Mixture-of-Experts block diffusion + latent Transformer-in-Transformer |64| Total Parameters | 26.139B |65| Activated Parameters | 4.729B, including the vision encoder |66| Text Heavy-Denoise Activated Parameters | 4.159B |67| FLOPs per Heavy Denoise | ~2.12 TFLOPs at batch 1, 256-token canvas, empty KV prefix |68| Layers | 30 |69| Number of Experts | 128 |70| Selected Experts per Token | 8 |71| Number of Shared Experts | 1 |72| Vocabulary Size | 262,144 |73| Context Length | 262,144 tokens |74| Activation Function | GELU, tanh approximation |75| Vision Encoder | Gemma 4 Vision |76| Vision Encoder Parameters | 569.550M |77| Modality | Text, Image, Video |78| Sliding Window | 1024 tokens |79| Canvas Length | 256 |80| Latent Memory | 64 slots × 1,536-d, 4 layers |81| Training tokens | ~7 million |82 83The heavy-denoise FLOPs estimate counts multiply-adds as two FLOPs and covers decoder, expert, latent deliberation, and attention work only. It excludes the encoder pass, sampling/softmax, and elementwise ops. Batch size scales it roughly linearly: a 256-token prefix raises the estimate to ~2.16 TFLOPs, and a 4,096-token prefix to ~2.38 TFLOPs because some layers use full attention.84 85## Benchmark Results86 87| Benchmark | Modilify Mk1 | DiffusionGemma 26B A4B | Gemma 4 26B A4B |88| --- | ---: | --- | --- |89| MMLU Pro | 86.8 | 77.6 | 82.6 |90 91Only part of each dataset was evaluated, with one-shot prompting. Treat these values as unstable and non-comparable until the full benchmark release.92 93## Getting Started94 95Transformers 5.14.1 is the minimum supported version.96 97```shell98pip install -U transformers torch accelerate99```100 101### Text generation102 103```python104import torch105from transformers import AutoModelForMultimodalLM, AutoProcessor106 107model_id = "modilify/Modilify-Mk1"108processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)109model = AutoModelForMultimodalLM.from_pretrained(110 model_id,111 trust_remote_code=True,112 dtype=torch.bfloat16,113 device_map="auto",114)115 116messages = [{"role": "user", "content": "Explain why the sky is blue."}]117inputs = processor.apply_chat_template(118 messages,119 tokenize=True,120 add_generation_prompt=True,121 enable_thinking=False,122 return_dict=True,123 return_tensors="pt",124).to(model.device)125 126output = model.generate(**inputs, max_new_tokens=256)127new_tokens = output.sequences[:, inputs["input_ids"].shape[1]:]128print(processor.batch_decode(new_tokens, skip_special_tokens=False)[0])129```130 131### Image input132 133```python134from PIL import Image135 136image = Image.open("example.jpg").convert("RGB")137messages = [{138 "role": "user",139 "content": [140 {"type": "image", "image": image},141 {"type": "text", "text": "Describe the image and identify uncertainty."},142 ],143}]144inputs = processor.apply_chat_template(145 messages,146 tokenize=True,147 add_generation_prompt=True,148 enable_thinking=True,149 return_dict=True,150 return_tensors="pt",151).to(model.device)152output = model.generate(**inputs, max_new_tokens=256)153```154 155### Video-frame input156 157The processor represents video as a sampled sequence of frames. The following example uses PyAV to decode a short local clip and samples at most 32 RGB frames.158 159```python160import av161from PIL import Image162 163container = av.open("short_clip.mp4")164decoded = [Image.fromarray(frame.to_rgb().to_ndarray()) for frame in container.decode(video=0)]165stride = max(1, len(decoded) // 32)166frames = decoded[::stride][:32]167 168messages = [{169 "role": "user",170 "content": [171 {"type": "video", "video": frames},172 {"type": "text", "text": "Summarize the main visual events in order."},173 ],174}]175inputs = processor.apply_chat_template(176 messages,177 tokenize=True,178 add_generation_prompt=True,179 enable_thinking=True,180 return_dict=True,181 return_tensors="pt",182).to(model.device)183output = model.generate(**inputs, max_new_tokens=256)184```185 186## Thinking mode187 188The official Gemma chat template controls the prompt, not the model's first generated tokens.189 190- `enable_thinking=True` inserts a system turn that contains `<|think|>` and still ends the prompt at `<|turn>model`.191- `enable_thinking=False` does **not** inject an empty thought channel. The prompt ends at `<|turn>model`.192 193The model may still open `<|channel>thought` on its own. That is generation, not a template artifact. Applications should not assume hidden reasoning is complete, correct, or appropriate to expose to end users.194 195## Configurable inference parameters196 197All model-owned values below are serialized in `config.json` and may be changed before loading or through a copied configuration object.198 199| Parameter | Default | Meaning |200| --- | ---: | --- |201| `canvas_length` | 256 | Rolling diffusion canvas length |202| `denoise_temperature` | 0.8 | Sampling temperature |203| `commit_failure_budget` | 0.2 | Normal cumulative prefix risk limit |204| `jump_failure_budget` | 2.0 | Forced-jump cumulative risk limit |205| `jump_on_no_progress_after` | 12 | Stagnation threshold |206| `max_ponder_steps` | 64 | Watchdog multiplier per requested token |207| `min_trajectory_progress` | 0.005 | Minimum fused-risk improvement |208| `repetition_penalty` | 1.0 | Transformers-style repetition penalty |209| `latent_dim` | 1,536 | Latent state width |210| `latent_memory_slots` | 64 | Persistent memory slot count |211| `latent_num_layers` | 4 | Latent Transformer depth |212| `latent_num_heads` | 16 | Latent attention heads |213| `latent_local_attention_window` | 128 | Local token-attention window |214| `latent_dropout` | 0.0 | Inference dropout probability |215| `turn_end_token_id` | 106 | Gemma turn terminator |216 217Example override:218 219```python220from transformers import AutoConfig221 222config = AutoConfig.from_pretrained(model_id, trust_remote_code=True)223config.max_ponder_steps = 32224config.commit_failure_budget = 0.15225model = AutoModelForMultimodalLM.from_pretrained(226 model_id,227 config=config,228 trust_remote_code=True,229 dtype=torch.bfloat16,230 device_map="auto",231)232```233 234Generation supports left-padded batches with independent stopping and `generated_lengths` for every row. Batch prompts of similar lengths together for the best throughput; KV-cache and canvas memory grow with batch size. Streaming and caller-supplied KV caches remain limited to batch size 1.235 236## Details237 238Trained on a single Apple silicon machine, in less than 24 hours, on about 7 million tokens.239 240Developed on Mac by Modilify.241 242## Evaluation status, limitations, and risks243 244The benchmark values above are partial one-shot estimates, not a complete evaluation. Export checks established checkpoint structure, exact adapter application, valid safetensors indexing, absence of residual adapters, and byte-level preservation of the vision tower and projection; they do not establish accuracy, robustness, calibration, fairness, safety, or fitness for use.245 246The model can hallucinate facts, citations, visual details, or temporal relationships; reproduce bias, unsafe content, personal information, or copyrighted material; and consume substantial time and memory during long iterative generation. Confidence-based commits are compute-control decisions, not guarantees of correctness. Visual performance can degrade with poor resolution, motion, occlusion, unusual aspect ratios, or domain shift.247 248Evaluate the exact deployment on representative, adversarial, and out-of-distribution inputs. Use layered safeguards, monitoring, incident response, and qualified human review, and never delegate autonomous high-risk medical, legal, financial, employment, housing, education, critical-infrastructure, or safety decisions to the model.249 250## License251 252Released under the [Modilify Open Model License 1.0](LICENSE), subject to its responsible-use and derivative-impact terms. Upstream rights, attribution, Apache-2.0 text, and the impact-statement template are retained in [NOTICE.md](NOTICE.md).253 254## Citation255 256```bibtex257@software{modilify_mk1_2026,258 title = {Modilify Mk1}, 259 author = {Modilify},260 year = {2026},261 note = {A multimodal latent-deliberation derivative of DiffusionGemma, trained on one Apple silicon machine}262}263```264 265Also cite the upstream DiffusionGemma release as requested by Google DeepMind.266