fernandofernandes/fly-connectome-49k

Fly Connectome 49k

The 49,393-neuron / 9,050,172-edge central-brain graph that `ngxson/fly-llm-hf` uses as its recurrent layer, packaged as plain typed arrays and joined to the MaleCNS body IDs, cell types, superclasses and soma positions it was derived from.

Until now this graph was reachable only by loading a 284 MB model checkpoint and reading its frozen buffers. This dataset is that graph, verifiable on its own, so you can build on the wiring without adopting anyone's model.

Looking for the whole fly? This is a central-brain subset. The complete MaleCNS v1.0 connectome — 166,700 neurons, 25,582,938 edges, 124,177,617 synaptic contacts, raw signed weights — is fly-connectome-malecns-166k, in the same conventions so the two compose.

This is measured biological connectivity, not a trained artifact. No weight here was learned. It is a scaled copy of the MaleCNS v1.0 connectome.

Contents — 76.13 MB

6,061,699 edges are positive, 2,988,473 negative, none zero. 44,279 of 49,393 neurons have a measured soma location; the remaining 5,114 are flagged in soma_valid and their coordinates are zeros, not estimates. 8,156 neurons carry an official cell type; the 1,005 without one each get their own group rather than being merged.

Load it

import json, numpy as np
from pathlib import Path
from huggingface_hub import snapshot_download

root = Path(snapshot_download("fernandofernandes/fly-connectome-49k", repo_type="dataset"))
manifest = json.loads((root / "manifest.json").read_text())

offsets = np.fromfile(root / "graph/edges_offsets.i32", dtype=np.int32)   # 49,394
source  = np.fromfile(root / "graph/edges_source.u16",  dtype=np.uint16)  # 9,050,172
weight  = np.fromfile(root / "graph/edges_weight.f32",  dtype=np.float32) # 9,050,172

# Incoming synapses of neuron 12345, as (presynaptic index, weight) pairs:
lo, hi = offsets[12345], offsets[12346]
incoming = list(zip(source[lo:hi], weight[lo:hi]))

As a SciPy sparse matrix, W[destination, source]:

from scipy.sparse import csr_matrix
W = csr_matrix((weight, source.astype(np.int32), offsets), shape=(49393, 49393))

The int16 encoding is value = code * manifest["weights"]["quantised"]["scale"]. It preserves every sign, with maximum relative error 1.53e-05. Use .f32 for anything numerical; .i16 exists to halve a browser download.

Verify it

Every file's SHA-256 is in manifest.json, and the manifest also carries the frozen-buffer digests recorded inside the trained model checkpoints — so you can prove this package is the same graph those models ran on, without downloading them. `verify_connectome_package.py` runs 18 such checks with nothing but NumPy:

All 18 checks passed: 49,393 neurons, 9,050,172 edges, 44,279 positioned somata, 9,161 cell-type groups.

Training against it

groups/groups.npz carries the verified neuron→cell-type grouping bound to this exact graph (node_type_index, body IDs, labels, and the frozen-buffer digests). It exists so that training a model on this connectome does not require re-deriving the grouping from the ~1 GB of upstream MaleCNS feather files. The fly-wordbrain pipeline fetches it straight from here:

python scripts/run_pipeline.py --quick     # one command, Apple Silicon

Provenance, and one honest gap

Derived from MaleCNS v1.0 central brain — superclasses cb_sensory, visual_projection, cb_intrinsic, ascending_neuron, descending_neuron — via the pinned reference checkpoint at revision 65c677b3d566a2e9793d5f72999cdb441c6c0a9f.

Every edge was matched back to its source endpoints, sign and globally scaled magnitude, with maximum absolute discrepancy 2.98e-08. The global scale from source synapse counts to checkpoint weights is 9.529943345114589e-04.

The gap: the induced source subgraph over these same 49,393 neurons contains 9,679,074 edges, but the checkpoint stores 9,050,172. The 628,902 missing edges were not restored, and we do not know why the original packaging omitted them. This dataset reproduces the checkpoint's topology exactly, because that is what the models were trained on — it is not a complete induced subgraph of MaleCNS. If you need completeness, go to the source data, not to this file.

Weights are scaled synapse counts with a transmitter-derived sign. They are not conductances or measured physiological strengths.

Licence and attribution

CC BY 4.0. Connectome data: FlyEM / HHMI Janelia Research Campus, University of Cambridge, MRC Laboratory of Molecular Biology, and Google Research. Graph packaging and interface layout follow `ngxson/fly-llm-hf` (CC BY 4.0). Packaging code is MIT.

Used by fly-wordbrain.