FlyinGodzilla/AMOR-GatedDeltaNet-180M

[image]

Paper · Code · Files · All models

[image]

Block 0 receives the backbone's final normalized output. Blocks 1 and 2 apply their own pre-normalization. A shared, embedding-tied LM head supplies all three entropy evaluations and the final output prediction.

A complete recurrent backbone, refined by three entropy-gated attention blocks.

Looking for the larger model? AMOR-Gated DeltaNet 1.5B.

The idea

AMOR (Adaptive Metacognitive Output Router) uses the model's predictive uncertainty to decide where attention should refine its recurrent representation. Each appended block reads normalized output entropy, compares it with a frozen threshold at inference, and adds an attention update only where its gate fires.

The recurrent backbone and attention blocks are pretrained jointly from scratch. “Post-hoc” describes where the attention blocks sit, after the complete backbone. Keys and values come from residual-stream projections and are retained at every position for future retrieval.

This model uses the native entropy gate. Optional distilled deployment routers are not included in this release. Training uses dense attention with an output mask, so gating does not reduce training attention FLOPs.

This model

Detailed settings are in training_config.json; training_summary.json records the completed run.

Results

This model's canonical eight-task common-sense mean is 35.7%. The score uses raw accuracy on LAMBADA, HellaSwag, PIQA, ARC-Easy, ARC-Challenge, WinoGrande, OpenBookQA and TruthfulQA-mc2.

Per-task entries are point estimates; the figure reports evaluation-repeat spread.

[image]

Research comparison across all three scales; use the 180M panel for this model. Whiskers show the sample SD of five reevaluations of each fixed model, centered on its canonical score. They measure evaluation repeatability, not independent-training variation. Axes are zoomed per panel.

Per-task values are also available in benchmark_results.json.

Download and run

These are custom PyTorch models. Use the supplied loader and generation script; this repository is not a Transformers AutoModel.from_pretrained integration, a hosted chat endpoint, or a GGUF/PocketPal package.

A Linux machine with a compatible NVIDIA GPU is the intended accelerated setup. Install a CUDA-enabled PyTorch build appropriate for your driver first (PyTorch installation). Then, in a fresh Python environment:

python -m pip install huggingface_hub
hf download FlyinGodzilla/AMOR-GatedDeltaNet-180M --local-dir AMOR-GatedDeltaNet-180M
cd AMOR-GatedDeltaNet-180M
python -m pip install "transformers==5.0.0" safetensors einops
python -m pip install flash-linear-attention
python generate.py --prompt "The Eiffel Tower is" --max-new-tokens 32

For the tested CPU reference path, use PyTorch 2.6.0 and Transformers 4.51.3. Mamba2 has a slow CPU fallback (--device cpu); Gated DeltaNet requires its compatible accelerated kernels. GPU kernel versions must match the PyTorch/CUDA environment. CUDA smoke tests used PyTorch 2.6.0+cu124, Transformers 5.0.0, Triton 3.2.0 and safetensors 0.7.0. Gated DeltaNet kernels: the imported fla module reported version 0.4.2. These are actual imported versions from the tested environment; the installation commands above have not been checked in a fresh environment.

generate.py performs cached greedy text completion. It is intentionally a plain-prompt example: these models were not trained as chat assistants. The tokenizer is referenced from teknium/Llama-3.1-AlternateTokenizer and pinned in config.json to the release-packaging revision. The original training-time tokenizer revision was not recorded.

Verification

CUDA strict loading, finite forward/prefill, cached decoding and the published generation command passed on an NVIDIA H100 NVL. The short cached/full-forward check had a maximum absolute logit difference of 0.0625 (rtol 0.02, atol 0.1). These are smoke tests, not a benchmark rerun.

Every exported tensor was compared exactly with the original trained weights, including the entropy gate's frozen buffers. The tied embedding/head matrix is stored once and restored by the loader. Optimizer state is excluded. Export metadata records provenance and actual validation; manifest.json lists file hashes.

Model family

All six models use the three-block canonical AMOR layout. Choose a backbone and scale to match your experiment; larger models require more memory. The cards use a snapshot of the research figures and do not automatically track later manuscript revisions.

Intended use and limitations

Research on recurrent–attention hybrids, uncertainty-based routing, and pretrained language-model behavior. Outputs can be incorrect and may inherit training-data biases. Entropy measures predictive uncertainty; it cannot detect every confidently wrong prediction. Inference fire rates depend on the workload, and attention KV storage still grows with context. Training context is 3,072 tokens; long-context experiments do not establish reliable performance on every domain.

License

The code and model weights are released under the MIT license; see LICENSE.

Citation

@article{zheng2026amor,
  title={When to Think Fast and Slow? AMOR: Adaptive Entropy Gate for Hybrid Models},
  author={Zheng, Haoran and Shani, Chen},
  journal={arXiv preprint arXiv:2602.13215},
  year={2026}
}