igorktech/nanofly-decoder-en

1---2license: cc-by-4.03language:4- en5library_name: transformers6pipeline_tag: text-generation7datasets:8- roneneldan/TinyStories9tags:10- connectome11- reservoir-computing12- echo-state-network13- fruit-fly14- drosophila15- malecns16- tinystories17- custom_code18---19 20# nanofly-decoder-en21 22![MaleCNS soma cloud coloured by this model's neuron states](connectome.png)23 24<sub>43,993 neurons at their measured MaleCNS v1.0 coordinates, coloured by this checkpoint's state at one tick while writing *"…there was a girl named Lily."* Orange excited, blue inhibited, grey at rest. Frontal view; the optic lobes flank the central brain.</sub>25 26A language model whose recurrent layer is the measured wiring of a fruit fly. The connectome is a frozen [echo state network](https://en.wikipedia.org/wiki/Echo_state_network) reservoir — **no synapse is trained**. Only the input projection, per-neuron gain/bias/leak, one global scale and the readout learn.27 28- **Code:** [github.com/igorktech/nanoFLY](https://github.com/igorktech/nanoFLY)29- **Russian sibling:** [`igorktech/nanofly-decoder-ru`](https://huggingface.co/igorktech/nanofly-decoder-ru)30 31## Usage32 33```python34import torch35from transformers import AutoTokenizer, AutoModelForCausalLM36 37repo = "igorktech/nanofly-decoder-en"38tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)39model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True).eval()40model = model.to("cuda" if torch.cuda.is_available() else "cpu")41 42ids = tok("Once upon a time, there was a", return_tensors="pt").input_ids43ids = torch.cat([torch.tensor([[model.config.bos_token_id]]), ids], dim=1).to(model.device)44out = model.generate(ids, max_new_tokens=80, do_sample=True, top_k=50, temperature=0.7)45print(tok.decode(out[0], skip_special_tokens=True))46```47 48- Prepend BOS: every training example started with it.49- Greedy: `do_sample=False`, and drop `top_k` or `transformers` warns.50- Beam search and assisted generation are unsupported (stateful model). Greedy, sampling, top-k, top-p work.51- ~960 forward passes/s on an RTX 5080, ~10/s on a laptop CPU.52 53## Architecture54 55| | |56|---|---|57| connectome | MaleCNS v1.0 central brain — `cb_sensory`, `cb_intrinsic`, `visual_projection`, `descending_neuron`, `ascending_neuron` |58| neurons / edges | 49,393 / 9,055,280 signed (623,728 dropped: modulatory or unknown transmitter) |59| edge weight | sign of the presynaptic transmitter × synapse count, rows normalised to unit absolute weight. ACh +1; GABA, Glu, His −1; others 0 |60| token input | 11,434 sensory-facing neurons, 8-slot delay line (slot *j* gets token *t−j*). No attention, no positional encoding |61| held out | the 2,635 ORNs stay out of the token input, so the encoder-decoder variant can start from these weights |62| dynamics | `x ← (1−a)·x + a·tanh(ρ·g·(Wx) + u + b)`, 2 ticks per token; `a` learned per neuron (init 0.5), `ρ` learned global (init 1.0 → **4.26**) |63| readout | all 49,393 states → `Linear(49393→256)` → `LayerNorm` → `Linear(256→1024)` |64| trainable | **16.25M** — readout 12.90M, input projection 2.93M, embedding 0.26M, per-neuron scalars 0.15M |65 66## Training67 68| | |69|---|---|70| data | TinyStories, 98,024 stories / 1,976 held out, 28,417,062 tokens, byte-level BPE vocab 1,024 |71| objective | next-token cross entropy, truncated BPTT over 32-token windows, state carried across windows |72| optimiser | AdamW — body 2e-3 (no decay), readout 5e-4 (decay 0.01), warmup 200 then cosine to 10%, clip 1.0 |73| schedule | 3 epochs, 22,115 updates, batch 128 |74| hardware | 1 × RTX 5080, 1.74 h at ~13,700 tok/s |75 76## Evaluation77 78Held-out loss. Identical data, recipe, seed and budget; only the recurrent layer differs:79 80| recurrent layer | val loss | ppl | trainable |81|---|---|---|---|82| degree-matched shuffled wiring, frozen | 1.979 | 7.2 | 16.25M |83| **this model** — real connectome, frozen | **1.933** | **6.9** | 16.25M |84| real connectome, synapse strengths trained | 1.913 | 6.8 | 25.30M |85 86The control keeps every neuron's in- and out-degree, the transmitter signs and Dale's law, and randomises only which neuron pairs with which. Two results, each holding at all ten intermediate evaluations:87 88- **The fly's specific wiring is worth 0.046 nats** over a random graph with its degrees — and that margin grows through training (0.031 → 0.046).89- **Making all 9,055,280 synapse strengths trainable is worth 0.020 nats** on top — and that margin shrinks through training (0.032 → 0.020), at 4× the compute. It buys faster convergence more than a higher ceiling. Trained strengths stay close to the measured ones (Pearson r = 0.94).90 91One seed per condition, so treat the margins as indicative rather than significant. Validation was still improving at the end in all three runs; none is converged.92 93Greedy samples, prompt in bold:94 95> **Once upon a time, there was a** little girl named Lily. She loved to play outside in the sunshine. One day, she saw a big, red ball that was very pretty. She wanted to play with it, but it was too high for her to reach.96 97> **Tom and his dog** were best friends. They liked to play in the park. One day, they saw a big dog with a big bag.98 99## Limitations100 101- 16M trainable parameters over 85M tokens of children's stories. It writes about Lily and Tom and nothing else.102- 8-token delay line plus a short leaky recurrent memory. Loss stops improving with context by roughly position 32; names and objects drift within a paragraph.103- A `tanh` rate neuron is not a spiking model: no spikes, no synaptic delays, no neuromodulation — modulatory edges are removed outright.104- Central brain only; the optic lobes and ventral nerve cord of the 166,700-neuron CNS are absent.105- Synapse count is a proxy for strength, and rows are normalised. Neither is measured physiology — see the third row of the evaluation table for what happens when the strengths are fit to the task instead.106 107## Credits108 109- **Connectome:** MaleCNS v1.0 — FlyEM / HHMI Janelia, University of Cambridge, MRC LMB, Google Research. CC BY 4.0. The published buffers derive from that release; keep the attribution when redistributing.110- **Transmitter signs:** Shiu et al., *Nature* 2024.111- **Connectome as reservoir:** Costi, Hadjiivanov, Dold, Hale, Izzo, 2025.112- **Prior art:** [`ngxson/fly-llm-hf`](https://huggingface.co/ngxson/fly-llm-hf), whose graph subset this reproduces.113- **Data:** Eldan and Li, *TinyStories*, 2023.114 115Weights CC BY 4.0, matching the connectome. Modeling code Apache-2.0.116