modilify/Modilify-Mk1

[image]

Modilify Mk1

A 26B-A5B multimodal block-diffusion model that thinks in latent space, commits only when it is ready, and was trained on a single Apple silicon machine in less than a day.

Modilify Mk1 is not another long-context decoder with a bigger reasoning budget. It is a Transformer-in-Transformer: the heavy DiffusionGemma trunk still sees text, images, and video, while a recurrent latent deliberation stack compresses the entire chain of thought into a compact hidden trajectory. Visible tokens are no longer the only place intelligence can live. The exclusive excess-entropy commit formula decides, every denoise, how many tokens the model is allowed to lock in. Easy problems finish fast. Hard problems keep deliberating.

This official release is the first public Mk1 checkpoint. It is materially more stable than Modilify Mk1 Preview, restores the full vision tower, and ships default inference settings that run about 6× faster than autoregressive models.

Breakthroughs

Why this is different

Most reasoning models buy intelligence with more visible tokens. That is expensive, leaky, and hard to stop. Mk1 buys it with latent deliberation:

1. 1.Each heavy denoise still runs the 26B-A4B MoE trunk over a 256-token rolling canvas.
2. 2.A 4-layer latent Transformer reads the noisy canvas, confidence, entropy, and age, then updates per-token latents plus a 64-slot persistent memory.
3. 3.That compact state is mapped back through the frozen self-conditioning bridge and conditions the next heavy pass.
4. 4.The exclusive commit formula then locks a variable-length prefix. The memory slots do not shift. The thought continues even after the visible tokens have moved on.

The result is elastic inference. You can spend more heavy-denoise work on a hard agent turn, or commit more tokens per pass and finish sooner when the problem is easy. Throughput is a configuration choice, not a second model.

Efficiency

Mk1 is an argument about intelligence per parameter and intelligence per training token.

The released model activates 4.159B text parameters on a heavy denoise, plus the 570M vision encoder when images or video are present. The latent stack is small. The adaptation that produced this checkpoint ran on one Apple silicon machine, finished in under 24 hours, and saw about 7 million tokens. That is not a claim that data does not matter. It is a claim that a better architecture can extract more from each token and each watt.

Default inference is the fast operating point. Compared with the slower Preview evaluation settings (denoise_temperature=0.4, commit_failure_budget=0.05, jump_on_no_progress_after=32), the Mk1 defaults are built for about 6× higher throughput. Tighten the budget if you want Preview-like caution. Loosen it if you want the model to finish.

Model Summary

The heavy-denoise FLOPs estimate counts multiply-adds as two FLOPs and covers decoder, expert, latent deliberation, and attention work only. It excludes the encoder pass, sampling/softmax, and elementwise ops. Batch size scales it roughly linearly: a 256-token prefix raises the estimate to ~2.16 TFLOPs, and a 4,096-token prefix to ~2.38 TFLOPs because some layers use full attention.

Benchmark Results

Only part of each dataset was evaluated, with one-shot prompting. Treat these values as unstable and non-comparable until the full benchmark release.

Getting Started

Transformers 5.14.1 is the minimum supported version.

pip install -U transformers torch accelerate

Text generation

import torch
from transformers import AutoModelForMultimodalLM, AutoProcessor

model_id = "modilify/Modilify-Mk1"
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForMultimodalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.bfloat16,
    device_map="auto",
)

messages = [{"role": "user", "content": "Explain why the sky is blue."}]
inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    enable_thinking=False,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

output = model.generate(**inputs, max_new_tokens=256)
new_tokens = output.sequences[:, inputs["input_ids"].shape[1]:]
print(processor.batch_decode(new_tokens, skip_special_tokens=False)[0])

Image input

from PIL import Image

image = Image.open("example.jpg").convert("RGB")
messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": image},
        {"type": "text", "text": "Describe the image and identify uncertainty."},
    ],
}]
inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    enable_thinking=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)
output = model.generate(**inputs, max_new_tokens=256)

Video-frame input

The processor represents video as a sampled sequence of frames. The following example uses PyAV to decode a short local clip and samples at most 32 RGB frames.

import av
from PIL import Image

container = av.open("short_clip.mp4")
decoded = [Image.fromarray(frame.to_rgb().to_ndarray()) for frame in container.decode(video=0)]
stride = max(1, len(decoded) // 32)
frames = decoded[::stride][:32]

messages = [{
    "role": "user",
    "content": [
        {"type": "video", "video": frames},
        {"type": "text", "text": "Summarize the main visual events in order."},
    ],
}]
inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    enable_thinking=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)
output = model.generate(**inputs, max_new_tokens=256)

Thinking mode

The official Gemma chat template controls the prompt, not the model's first generated tokens.

• —enable_thinking=True inserts a system turn that contains <|think|> and still ends the prompt at <|turn>model.
• —enable_thinking=False does not inject an empty thought channel. The prompt ends at <|turn>model.

The model may still open <|channel>thought on its own. That is generation, not a template artifact. Applications should not assume hidden reasoning is complete, correct, or appropriate to expose to end users.

Configurable inference parameters

All model-owned values below are serialized in config.json and may be changed before loading or through a copied configuration object.

Example override:

from transformers import AutoConfig

config = AutoConfig.from_pretrained(model_id, trust_remote_code=True)
config.max_ponder_steps = 32
config.commit_failure_budget = 0.15
model = AutoModelForMultimodalLM.from_pretrained(
    model_id,
    config=config,
    trust_remote_code=True,
    dtype=torch.bfloat16,
    device_map="auto",
)

Generation supports left-padded batches with independent stopping and generated_lengths for every row. Batch prompts of similar lengths together for the best throughput; KV-cache and canvas memory grow with batch size. Streaming and caller-supplied KV caches remain limited to batch size 1.

Details

Trained on a single Apple silicon machine, in less than 24 hours, on about 7 million tokens.

Developed on Mac by Modilify.

Evaluation status, limitations, and risks

The benchmark values above are partial one-shot estimates, not a complete evaluation. Export checks established checkpoint structure, exact adapter application, valid safetensors indexing, absence of residual adapters, and byte-level preservation of the vision tower and projection; they do not establish accuracy, robustness, calibration, fairness, safety, or fitness for use.

The model can hallucinate facts, citations, visual details, or temporal relationships; reproduce bias, unsafe content, personal information, or copyrighted material; and consume substantial time and memory during long iterative generation. Confidence-based commits are compute-control decisions, not guarantees of correctness. Visual performance can degrade with poor resolution, motion, occlusion, unusual aspect ratios, or domain shift.

Evaluate the exact deployment on representative, adversarial, and out-of-distribution inputs. Use layered safeguards, monitoring, incident response, and qualified human review, and never delegate autonomous high-risk medical, legal, financial, employment, housing, education, critical-infrastructure, or safety decisions to the model.

License

Released under the Modilify Open Model License 1.0, subject to its responsible-use and derivative-impact terms. Upstream rights, attribution, Apache-2.0 text, and the impact-statement template are retained in NOTICE.md.

Citation

@software{modilify_mk1_2026,
  title = {Modilify Mk1}, 
  author = {Modilify},
  year = {2026},
  note = {A multimodal latent-deliberation derivative of DiffusionGemma, trained on one Apple silicon machine}
}

Also cite the upstream DiffusionGemma release as requested by Google DeepMind.