QuixiAI/FlyGPT

1---2license: cc-by-4.03language: [en]4library_name: transformers5pipeline_tag: text-generation6base_model: QuixiAI/MaleCNS7base_model_relation: finetune8datasets: [karpathy/tiny_shakespeare]9tags: [connectome, fruit-fly, drosophila, malecns, recurrent, sparse, tiny-shakespeare, custom_code]10---11 12# QuixiAI/FlyGPT13 14A character-level language model whose recurrent architecture is **a real subgraph of the fruit-fly brain15connectome** ([MaleCNS v1.0](https://male-cns.janelia.org/)). Unlike the earlier frozen-reservoir approach in16[ngxson/fly-llm-hf](https://huggingface.co/ngxson/fly-llm-hf), which keeps the connectome's synaptic weights fixed and17trains only the projections and readout, FlyGPT **trains one value per real synaptic connection with gradient18descent** while keeping the fly's edge topology fixed, and compares the result against the same neurons with19degree-preserving scrambled connections across paired seeds.20 21**Base model: [QuixiAI/MaleCNS](https://huggingface.co/QuixiAI/MaleCNS)**, the lossless packaging of the MaleCNS v1.022connectivity tables. FlyGPT's graph is extracted from it deterministically (`build_graph.py`, revision pinned in23`data/fly/build_edges.py`); `graph.node_id` and `graph.synapse_count` map every edge back to that repository.24 25This checkpoint's wiring is the original MaleCNS wiring.26 27**Trained.** Condition `real`, seed 1, step 94000, validation loss 1.5778 nats/char on the fixed Tiny Shakespeare split.28 29This is not a biological simulation of a living fly. The "weights" in the MaleCNS release are anatomical synapse30counts; they are stored here as `graph.synapse_count` and are **not** the model's parameters.31 32## The graph33 34Every number below is produced by FlyGPT's extraction script (`build_graph.py`), not typed by hand.35 36| | |37|---|---|38| Source | MaleCNS v1.0 flat connectome (`gs://flyem-male-cns/v1.0/connectome-data/flat-connectome/`) |39| Candidate pool | central brain: `superclass` starting with `cb_` (37,108 neurons) |40| Minimum synapses per connection | 3 (engineering choice, not a biological claim) |41| Extraction | largest SCC → largest directed (k,k)-core with ≥ target nodes (k = 40) → trim by weighted degree |42| Neurons used | 5,000 |43| Directed connections used | 524,324 |44| Synaptic contacts represented | 8,300,915 |45| Largest SCC fraction | 1.0 |46| Reciprocal pairs | 93,055 |47| Input / output neurons | top 256 by out-degree / top 512 by in-degree |48| Input→output shortest path (median / p90 / max hops) | 1.0 / 1.0 / 1.0 |49| Graph hash | `f82b783b7ccb5a354fc4cf3de6de4a98d75029303c55f8faae28ab807828a007` |50 51`graph.node_id` holds the MaleCNS body ids, so every neuron maps back to the release.52 53## What is in `model.safetensors`54 55| tensor | shape | dtype | size |56|---|---|---|---|57| `graph.edge_index` | (2, 524324) | int32 | 4.19 MB |58| `graph.synapse_count` | (524324,) | int32 | 2.10 MB |59| `graph.node_id` | (5000,) | int64 | 0.04 MB |60| `graph.input_nodes` | (256,) | int64 | 0.00 MB |61| `graph.output_nodes` | (512,) | int64 | 0.00 MB |62| `recurrent.edge_values` | (524324,) | bfloat16 | 1.05 MB |63| `recurrent.bias` | (5000,) | bfloat16 | 0.01 MB |64| `recurrent.raw_leak` | (5000,) | bfloat16 | 0.01 MB |65| `embed.weight` | (65, 32) | bfloat16 | 0.00 MB |66| `input_proj.weight` | (256, 32) | bfloat16 | 0.02 MB |67| `input_proj.bias` | (256,) | bfloat16 | 0.00 MB |68| `lm_head.weight` | (65, 512) | bfloat16 | 0.07 MB |69| `lm_head.bias` | (65,) | bfloat16 | 0.00 MB |70 71`graph.*` is the anatomy (integer, never trained). `recurrent.*`, `embed.*`, `input_proj.*`, `lm_head.*` are the72learned state, stored in bf16. The sparse recurrent matmul is rebuilt in fp32 at runtime (rows = destination,73columns = source), with each incoming edge scaled by `1/sqrt(in_degree)`.74 75## Dynamics76 77```text78character → embedding (32) → linear → 256 input neurons79proposal_i = tanh( Σ_j W_ij h_j / sqrt(in_degree_i) + external_input_i + bias_i )80h_i ← (1 − leak_i) h_i + leak_i · proposal_i        (2 microsteps per character, leak_i = sigmoid(raw_leak_i))81512 output neuron states → linear → 65 logits82```83 84## Inference85 86```python87import torch88from transformers import AutoModelForCausalLM, AutoTokenizer89 90tok = AutoTokenizer.from_pretrained("QuixiAI/FlyGPT")91model = AutoModelForCausalLM.from_pretrained("QuixiAI/FlyGPT", trust_remote_code=True, dtype=torch.float32)92 93ids = tok("ROMEO:", return_tensors="pt").input_ids94out = model.generate(ids, max_new_tokens=300, do_sample=True, temperature=0.8)95print(tok.decode(out[0]))96 97# The degree-preserving scrambled control (same neurons, same degrees, shuffled wiring), for comparison:98scrambled = AutoModelForCausalLM.from_pretrained("QuixiAI/FlyGPT", subfolder="scrambled", trust_remote_code=True, dtype=torch.float32)99print(tok.decode(scrambled.generate(ids, max_new_tokens=300, do_sample=True, temperature=0.8)[0]))100 101# Neuron activity, for visualization: [1, T, 5000] states after each character, plus MaleCNS body ids102with torch.no_grad():103    states = model(ids).state                                   # [B, N] after the last character104body_ids = model.graph.node_id                                  # index -> MaleCNS body id, for lookup in QuixiAI/MaleCNS105```106 107The tokenizer is strict: only the 65 characters of Tiny Shakespeare are encodable. `generate()` carries the neuron108state between characters instead of a KV cache.109 110## Training111 112The recurrent core has one trainable weight per real synaptic connection. With the113[connectome-kernels](https://github.com/QuixiAI/connectome-kernels) package installed, the model's forward pass114runs on fused CUDA kernels (about 13× faster than `torch.sparse`, identical gradients); without it, it falls back115to `torch.sparse` automatically.116 117```python118# Fine-tune / continue training FlyGPT on Tiny Shakespeare (character-level).119# pip install transformers safetensors120# pip install --no-build-isolation git+https://github.com/QuixiAI/connectome-kernels   # fused CUDA path, ~13x faster121import requests, torch, torch.nn.functional as F122from transformers import AutoModelForCausalLM, AutoTokenizer123 124tok = AutoTokenizer.from_pretrained("QuixiAI/FlyGPT")125model = AutoModelForCausalLM.from_pretrained("QuixiAI/FlyGPT", trust_remote_code=True, dtype=torch.float32).cuda()126# start from the untrained initialization instead:  subfolder="init"127 128text = requests.get("https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt").text129data = torch.tensor(tok(text).input_ids)130train, val = data[: int(0.9 * len(data))], data[int(0.9 * len(data)):]   # FlyGPT's fixed 90/10 split131 132def batch(split, B=32, T=64):133    i = torch.randint(0, len(split) - T - 1, (B,))134    x = torch.stack([split[j : j + T] for j in i]); y = torch.stack([split[j + 1 : j + T + 1] for j in i])135    return x.cuda(), y.cuda()136 137recurrent = list(model.recurrent.parameters())                         # one weight per real synapse, bias, leak138adapters = [p for n, p in model.named_parameters() if not n.startswith("recurrent.")]139opt = torch.optim.AdamW([{"params": adapters, "lr": 1e-3}, {"params": recurrent, "lr": 3e-4}], weight_decay=0.01)140 141for step in range(1, 501):142    x, y = batch(train)143    logits = model(x).logits                                           # [B, T, 65]; state resets to zero per window144    loss = F.cross_entropy(logits.reshape(-1, 65), y.reshape(-1))145    opt.zero_grad(set_to_none=True); loss.backward()146    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0); opt.step()147    if step % 100 == 0:148        with torch.no_grad():149            vx, vy = batch(val); vl = F.cross_entropy(model(vx).logits.reshape(-1, 65), vy.reshape(-1))150        print(f"step {step}  train {loss.item():.3f}  val {vl.item():.3f}")151 152model.save_pretrained("flygpt-finetuned"); tok.save_pretrained("flygpt-finetuned")153```154 155## Result: does the wiring matter?156 157Pre-registered rule (before any result was seen): "wiring matters" is claimed only if all 5 paired158differences Δ = loss(scrambled) − loss(real) have the same sign **and** the mean Δ is at least 0.05 nats/char.159 160| seed | real | degree-preserving scramble | Δ |161|--:|--:|--:|--:|162| 1 | 1.6044 | 1.6125 | +0.0081 |163| 2 | 1.6193 | 1.6299 | +0.0106 |164| 3 | 1.6031 | 1.6064 | +0.0033 |165| 4 | 1.6083 | 1.6065 | -0.0018 |166| 5 | 1.6114 | 1.6073 | -0.0041 |167| **mean** | **1.6093** | **1.6125** | **+0.0032** |168 169Validation loss in nats/char at the end of 100,000 steps, same data order, batches, adapter init and edge-value RNG170stream per seed. Bigram reference on this split: 2.482. The differences are small and not all of the same sign, so the171verdict is: **no detectable difference at this scale.** The fly connectome learns Shakespeare; at 5,000 neurons its specific wiring does not172measurably beat a degree-matched scramble. (At 20,000 steps the real wiring led on all five seeds by a mean of1730.011 nats; with full training the scramble catches up, so that early edge is a learning-speed effect.)174 175## Citation176 177If you use this model, please cite it, its base model, and the MaleCNS dataset paper.178 179This model:180 181```bibtex182@misc{hartford2026flygpt,183  title        = {FlyGPT: a language model whose recurrent architecture is a real subgraph of the fruit-fly connectome},184  author       = {Hartford, Eric},185  year         = {2026},186  publisher    = {Hugging Face},187  howpublished = {\url{https://huggingface.co/QuixiAI/FlyGPT}},188  note         = {Base model: QuixiAI/MaleCNS (MaleCNS v1.0, Berg et al. 2026, CC-BY 4.0). Code: https://github.com/QuixiAI/FlyGPT}189}190```191 192The base model (lossless connectome packaging):193 194```bibtex195@misc{hartford2026malecns,196  title        = {QuixiAI/MaleCNS: the MaleCNS v1.0 fruit-fly connectome as lossless Safetensors},197  author       = {Hartford, Eric},198  year         = {2026},199  publisher    = {Hugging Face},200  doi          = {10.57967/hf/10410},201  howpublished = {\url{https://huggingface.co/QuixiAI/MaleCNS}},202  note         = {Repackaging of Berg et al. (2026), CC-BY 4.0}203}204```205 206The dataset (required by the CC-BY 4.0 license):207 208```bibtex209@article{berg2026malecns,210  title     = {Sexual dimorphism in the complete {Drosophila} male central nervous system connectome},211  author    = {Berg, Stuart and Beckett, Isabella R. and Costa, Marta and Schlegel, Philipp and Januszewski, Michał and Marin, Elizabeth C. and Nern, Aljoscha and Preibisch, Stephan and Qiu, Wei and Takemura, Shin-ya and Fragniere, Alexandra M.C. and Champion, Andrew S. and Adjavon, Diane-Yayra and Cook, Michael and Gkantia, Marina and Hayworth, Kenneth J. and Huang, Gary B. and Katz, William T. and Kämpf, Florian and Lu, Zhiyuan and Ordish, Christopher and Paterson, Tyler and Stürner, Tomke and Trautman, Eric T. and Whittle, Catherine R. and Burnett, Laura E. and Hoeller, Judith and Li, Feng and Loesche, Frank and Morris, Billy J. and Pietzsch, Tobias and Pleijzier, Markus W. and Silva, Valeria and Yin, Yijie and Ali, Iris and Badalamente, Griffin and Bates, Alexander Shakeel and Beresford, Rory J. and Bogovic, John and Brooks, Paul and Cachero, Sebastian and Canino, Brandon S. and Chaisrisawatsuk, Bhumpanya and Clements, Jody and Crowe, Arthur and de Haan Vicente, Inês and Dempsey, Georgia and Donà, Erika and Dos Santos, Márcia and Dreher, Marisa and Dunne, Christopher R. and Eichler, Katharina and Finley-May, Samantha and Flynn, Miriam A. and Hameed, Imran and Hopkins, Gary Patrick and Hubbard, Philip M. and Kiassat, Ladann and Kovalyak, Julie and Lauchie, Shirley A. and Leonard, Meghan and Lohff, Alanna and Longden, Kit D. and Maldonado, Charli A. and Moitra, Ilina and Moon, Sung Soo and Mooney, Caroline and Munnelly, Eva J. and Okeoma, Nneoma and Olbris, Donald J. and Pai, Anika and Patel, Birava and Phillips, Emily M. and Plaza, Stephen M. and Richards, Alana and Rivas Salinas, Jennifer and Roberts, Ruairí J.V. and Rogers, Edward M. and Scott, Ashley L. and Scuderi, Louis A. and Seenivasan, Pavithraa and Serratosa Capdevila, Laia and Smith, Claire and Svirskas, Rob and Takemura, Satoko and Tastekin, Ibrahim and Thomson, Alexander and Umayam, Lowell and Walsh, John J. and Whittome, Holly and Xu, C. Shan and Yakal, Emily A. and Yang, Tansy and Zhao, Arthur and George, Reed and Jain, Viren and Jayaraman, Vivek and Korff, Wyatt and Meissner, Geoffrey W. and Romani, Sandro and Funke, Jan and Knecht, Christopher and Saalfeld, Stephan and Scheffer, Louis K. and Waddell, Scott and Card, Gwyneth M. and Ribeiro, Carlos and Reiser, Michael B. and Hess, Harald F. and Rubin, Gerald M. and Jefferis, Gregory S.X.E.},212  journal   = {Cell},213  volume    = {189},214  number    = {18},215  pages     = {5504--5526.e15},216  year      = {2026},217  month     = sep,218  publisher = {Elsevier},219  doi       = {10.1016/j.cell.2026.08.015},220  url       = {https://doi.org/10.1016/j.cell.2026.08.015},221  note      = {Preprint: bioRxiv 10.1101/2025.10.09.680999. Data: MaleCNS v1.0, CC-BY 4.0, https://male-cns.janelia.org}222}223```224 225## License226 227The connectome is released under CC-BY 4.0 by the FlyEM Project Team (HHMI Janelia), the University of Cambridge,228the MRC Laboratory of Molecular Biology, and Google Research. This checkpoint is a derivative and carries the same229license.230 231Prior art: [ngxson/fly-llm-hf](https://huggingface.co/ngxson/fly-llm-hf) (frozen MaleCNS reservoir LM) and232[eob/gpt-fly](https://huggingface.co/eob/gpt-fly) (FlyWire-masked GPT-2). Code and experiment:233[github.com/QuixiAI/FlyGPT](https://github.com/QuixiAI/FlyGPT).234