modilify/Modilify-Mk1-preview

1---2license: other3license_name: modilify-open-model-license-1.04license_link: LICENSE5library_name: transformers6pipeline_tag: image-text-to-text7tags:8  - diffusion9  - multimodal10  - image-text-to-text11  - mixture-of-experts12  - trust-remote-code13---14 15![LOGO](assets/01-LOGO.jpg)16 17# Modilify Mk1 Preview18 19Modilify Mk1 Preview is a 26B-A5B multimodal block-diffusion model developed by Modilify. It combines the original DiffusionGemma text, image, and video context encoder with recurrent latent deliberation and a multiplicative confidence-and-entropy commit policy. Its central advance is elastic inference: the model can spend more heavy-denoise work and produce a longer response for a difficult problem, or commit more tokens per pass and finish sooner when the problem is easier.20 21## Model Summary22 23| | |24| --- | ---: |25| Architecture | Mixture-of-Experts block diffusion |26| Total Parameters | 26.139B |27| Activated Parameters | 4.729B, including the vision encoder |28| Text Heavy-Denoise Activated Parameters | 4.159B |29| FLOPs per Heavy Denoise | ~2.12 TFLOPs at batch 1, 256-token canvas, empty KV prefix |30| Layers | 30 |31| Number of Experts | 128 |32| Selected Experts per Token | 8 |33| Number of Shared Experts | 1 |34| Vocabulary Size | 262,144 |35| Context Length | 262,144 tokens |36| Activation Function | GELU, tanh approximation |37| Vision Encoder | Gemma 4 Vision |38| Vision Encoder Parameters | 569.550M |39| Modality | Text, Image, Video |40| Sliding Window | 1024 tokens |41| Canvas Length | 256 |42 43The heavy-denoise FLOPs estimate counts multiply-adds as two FLOPs and covers decoder, expert, latent deliberation, and attention work only. It excludes the encoder pass, sampling/softmax, and elementwise ops. Batch size scales it roughly linearly: a 256-token prefix raises the estimate to ~2.16 TFLOPs, and a 4,096-token prefix to ~2.38 TFLOPs because some layers use full attention.44 45## Benchmark Results46 47| Benchmark | Modilify Mk1 Preview | Status |48| --- | ---: | --- |49| MMLU Pro | ~78.0 | Partial, one-shot estimate |50| AIME 2026, no tools | ~70 | Partial, one-shot estimate |51 52These preliminary results use specialized inference settings: `denoise_temperature=0.4`, `commit_failure_budget=0.05`, `fused_entropy_weight=0.6`, and `jump_on_no_progress_after=32`. The defaults target substantially higher throughput and should not be expected to reproduce these estimates.53 54Only part of each dataset was evaluated, with one-shot prompting. Treat these values as unstable and non-comparable until the full benchmark release.55 56## Core Capabilities57 58- **Elastic reasoning length and throughput.** Unlike a fixed reasoning-token budget, each heavy denoise can commit a variable-length prefix. Easier problems can converge in fewer passes with more tokens committed per pass; harder problems can keep deliberating and produce longer answers, up to the configured generation and ponder bounds. `commit_failure_budget`, `fused_entropy_weight`, and the progress limits let deployments move the throughput-quality operating point.59- **Implicit context through latent deliberation.** The model reasons over the complete noisy canvas inside each heavy denoise, then compresses the evolving trajectory into recurrent token latents and fixed memory slots. That compact state conditions the next heavy pass and survives rolling-canvas commits, providing an implicit context channel without copying the entire denoise history into visible output tokens.60- **120x faster trajectory training.** The current memory-bounded training system delivered a measured 120x throughput improvement over the project's original trajectory-training implementation under the matched internal setup. This figure describes the training implementation comparison, not a 120x inference speedup or a comparison with unrelated systems.61- Text generation supports the native optional thinking-channel protocol, and input context extends to the configured 262,144-token limit.62 63## Getting Started64 65Transformers 5.14.1 is the minimum supported version. To get started, install Transformers, PyTorch, and Accelerate in your environment:66 67```shell68pip install -U transformers torch accelerate69```70 71### Text generation72 73```python74import torch75from transformers import AutoModelForMultimodalLM, AutoProcessor76 77model_id = "modilify/Modilify-Mk1-preview"78processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)79model = AutoModelForMultimodalLM.from_pretrained(80    model_id,81    trust_remote_code=True,82    dtype=torch.bfloat16,83    device_map="auto",84)85 86messages = [{"role": "user", "content": "Explain why the sky is blue."}]87inputs = processor.apply_chat_template(88    messages,89    tokenize=True,90    add_generation_prompt=True,91    enable_thinking=False,92    return_dict=True,93    return_tensors="pt",94).to(model.device)95 96output = model.generate(**inputs, max_new_tokens=256)97new_tokens = output.sequences[:, inputs["input_ids"].shape[1]:]98print(processor.batch_decode(new_tokens, skip_special_tokens=False)[0])99```100 101### Image input102 103```python104from PIL import Image105 106image = Image.open("example.jpg").convert("RGB")107messages = [{108    "role": "user",109    "content": [110        {"type": "image", "image": image},111        {"type": "text", "text": "Describe the image and identify uncertainty."},112    ],113}]114inputs = processor.apply_chat_template(115    messages,116    tokenize=True,117    add_generation_prompt=True,118    enable_thinking=True,119    return_dict=True,120    return_tensors="pt",121).to(model.device)122output = model.generate(**inputs, max_new_tokens=256)123```124 125### Video-frame input126 127The processor represents video as a sampled sequence of frames. The following example uses PyAV to decode a short local clip and samples at most 32 RGB frames.128 129```python130import av131from PIL import Image132 133container = av.open("short_clip.mp4")134decoded = [Image.fromarray(frame.to_rgb().to_ndarray()) for frame in container.decode(video=0)]135stride = max(1, len(decoded) // 32)136frames = decoded[::stride][:32]137 138messages = [{139    "role": "user",140    "content": [141        {"type": "video", "video": frames},142        {"type": "text", "text": "Summarize the main visual events in order."},143    ],144}]145inputs = processor.apply_chat_template(146    messages,147    tokenize=True,148    add_generation_prompt=True,149    enable_thinking=True,150    return_dict=True,151    return_tensors="pt",152).to(model.device)153output = model.generate(**inputs, max_new_tokens=256)154```155 156## Thinking mode157 158The Gemma chat template controls thinking markup. Set `enable_thinking=True` in `apply_chat_template` to request the native thinking/channel protocol, or set it to `False` for a direct answer. Applications should not assume hidden reasoning is complete, correct, or appropriate to expose to end users.159 160## Configurable inference parameters161 162All model-owned values below are serialized in `config.json` and may be changed before loading or through a copied configuration object.163 164| Parameter | Default | Meaning |165| --- | ---: | --- |166| `canvas_length` | 256 | Rolling diffusion canvas length |167| `denoise_temperature` | 0.8 | Sampling temperature |168| `commit_failure_budget` | 0.2 | Normal cumulative prefix risk limit |169| `fused_entropy_weight` | 0.5 | Entropy multiplier coefficient |170| `jump_failure_budget` | 2.0 | Forced-jump cumulative risk limit |171| `vocab_chunk_size` | 65,536 | Serialized projection planning size; standard tensor operations are used |172| `jump_on_no_progress_after` | 12 | Stagnation threshold |173| `max_ponder_steps` | 64 | Watchdog multiplier per requested token |174| `min_trajectory_progress` | 0.005 | Minimum fused-risk improvement |175| `latent_dim` | 1,536 | Latent state width |176| `latent_memory_slots` | 64 | Persistent memory slot count |177| `latent_num_layers` | 4 | Latent Transformer depth |178| `latent_num_heads` | 16 | Latent attention heads |179| `latent_local_attention_window` | 128 | Local token-attention window |180| `latent_dropout` | 0.0 | Inference dropout probability |181| `turn_end_token_id` | 106 | Gemma turn terminator |182 183Example override:184 185```python186from transformers import AutoConfig187 188config = AutoConfig.from_pretrained(model_id, trust_remote_code=True)189config.max_ponder_steps = 32190config.commit_failure_budget = 0.15191model = AutoModelForMultimodalLM.from_pretrained(192    model_id,193    config=config,194    trust_remote_code=True,195    dtype=torch.bfloat16,196    device_map="auto",197)198```199 200Generation supports left-padded batches with independent stopping and `generated_lengths` for every row. Batch prompts of similar lengths together for the best throughput; KV-cache and canvas memory grow with batch size. Streaming and caller-supplied KV caches remain limited to batch size 1.201 202## Details203 204Trained on Apple silicon.205 206Developed on Mac by Modilify.207 208## Evaluation status, limitations, and risks209 210The benchmark values above are partial one-shot estimates, not a complete evaluation. Export checks established checkpoint structure, exact adapter application, valid safetensors indexing, absence of residual adapters, and byte-level preservation of the vision tower and projection; they do not establish accuracy, robustness, calibration, fairness, safety, or fitness for use.211 212The model can hallucinate facts, citations, visual details, or temporal relationships; reproduce bias, unsafe content, personal information, or copyrighted material; and consume substantial time and memory during long iterative generation. Confidence-based commits are compute-control decisions, not guarantees of correctness. Visual performance can degrade with poor resolution, motion, occlusion, unusual aspect ratios, or domain shift.213 214Evaluate the exact deployment on representative, adversarial, and out-of-distribution inputs. Use layered safeguards, monitoring, incident response, and qualified human review, and never delegate autonomous high-risk medical, legal, financial, employment, housing, education, critical-infrastructure, or safety decisions to the model.215 216## License217 218Released under the draft [Modilify Open Model License 1.0](LICENSE), subject to its responsible-use and derivative-impact terms. Upstream rights, attribution, Apache-2.0 text, and the impact-statement template are retained in [NOTICE.md](NOTICE.md). Obtain independent legal review before public release.219 220## Citation221 222Until a formal paper is available, cite the model repository and its base:223 224```bibtex225@software{modilify_mk1_preview_2026,226  title = {Modilify Mk1 Preview},227  author = {Modilify},228  year = {2026},229  note = {A multimodal latent-deliberation derivative of DiffusionGemma}230}231```232 233Also cite the upstream DiffusionGemma release as requested by Google DeepMind.234