SubstrateCommons/disco-replay

DiSCo as a replay phantom

The DiSCo substrate (Rafael-Patino, Girard, Truffet, Pizzolato, Caruyer, Thiran, The diffusion-simulated connectivity (DiSCo) dataset, Data in Brief 38 (2021) 107429, doi:10.1016/j.dib.2021.107429; data doi:10.17632/fgf86jdfg6.3, CC BY 4.0) walked once and stored as a replay pack, so that any acquisition a human scanner can play is a replay of the same walk, voxel by voxel on the dataset's own 40³ grid of 25 µm voxels, with a per-voxel Monte-Carlo certificate. DiSCo published one acquisition of this substrate; this dataset is the substrate itself in replayable form: the same 12,196 strands, two tubes per strand (the listed inner diameter and the outer tube at 1/0.7 of it), the dataset's diffusivity in both pools (0.6e-9 m²/s), intra water inside the inner tube and extra water outside the outer one, walked for 100 ms with every tier (positions, occupancy, wall contact, the strand field along the path).

Use me

Install dmipy-sim from its main branch (the reader lives there); the dataset is public, no login is needed:

pip install "git+https://github.com/dmrai-lab/dmipy-sim@main" "huggingface_hub>=0.25"

1. You want images and no compute. disco/reference/ holds pre-replayed volumes on the 40³ grid as S/S0 NIfTI with their gradient tables: DiSCo's own 364-measurement protocol as bare diffusion (the dataset's own simulation) and with white matter at 3 T and 7 T (T2 per pool, surface relaxivity, the sheath's susceptibility), four other acquisitions (a clinical b = 1000 shell, a three-shell research scheme, a Connectome 2.0 b = 6000 shell, a 50 Hz OGSE), the per-voxel floor, the record of what each was read with, the comparison with the dataset's own images, and tractography scores. Read its README.md first.

2. You want to replay your own acquisition. Open the pack by reference; nothing is downloaded whole, every byte read is one the replay uses (HTTP range reads of the columnar layout under disco/):

from dmipy_sim.replay import ReplayPack
from dmipy_sim import sequences
from dmipy_sim.spec.tissue import Tissue

pack = ReplayPack.open("hf://SubstrateCommons/disco-replay/disco")       # 150 M walkers, 31,802 voxels; ~0 bytes so far
seq = sequences.pgse([[1, 0, 0], [0, 0, 1]], 0.0102, 0.0167, bvalues=[1e9, 1e9], TE=0.0535)   # SI: s/m², s; any grid, any TE ≤ 100 ms

print(pack.plan(seq))                                                    # bands, tiers, bytes: decided before any transfer
view = pack.view(K=32, voxels=[(20, 20, 20)])                            # one voxel's rows at 32 bands (~1 MB): an ordinary ReplayPack
S = view.replay(seq)                                                     # bare diffusion, the dataset's own physics
wm = Tissue(T2={"intra": 0.05, "extra": 0.055, "myelin": 0.01}, rho=1.16e-6, chi_iso=-1e-7, chi_aniso=-1e-7)
view = pack.view(K=32, modes=8, contact=True, voxels=[(20, 20, 20)])     # the tiers the physics needs
S_3T = view.replay(seq, tissue=wm, scanner=3.0)                          # relaxation, wall contact, the sheath field at 3 T

from dmipy_sim.replay.study import Acquisition, Protocol, Study
study = Study(Protocol([Acquisition(seq)]), tissues=[None, wm], scanners=[None, 3.0, 7.0], pairs=[(0, 0), (1, 1), (1, 2)])
S, floor, plan = pack.image(study)                                       # the whole grid: one pass over the rows, every pair from it

S is (pairs, 40, 40, 40, measurements), NaN where the pack holds no walkers; floor the split-half floor of each volume, measured in the same pass. A study names a protocol (sequences, each in a pose), the tissues and the scanners; the bands are contracted once per acquisition and every tissue and scanner is arithmetic on the result. A replay returns the signal as measured (with the relaxation decay at TE when a tissue carries a T2); divide by a b = 0 measurement of the same setting for S/S0, as the reference volumes are. A pass over all rows streams 20 to 100 GB depending on the tiers (about 10 to 25 minutes at 50-75 MB/s) and needs a GPU for the sums; a voxel takes a second. The replay knobs are three objects, each stated once: tissue (a Tissue, or pack.nominal for the spec's values), scanner (a field in tesla or a catalogue scanner), orientation. Nothing is applied silently: the default is bare diffusion. The replay guide in dmipy-sim is the manual: one page per object, the table of what each knob touches and which channel it needs, images by reference, and a DiSCo recipe.

3. You want the raw packs. blocks/disco/block-NNNN.p1.rpk is the pass-1 shard of voxel block NNNN (plan/blocks.json: its i, j, k box); ReplayPack.load reads one, dmipy_sim.replay.bank.merge_packs joins several. The embedded spec cites the strand file at substrate/DiSCo_Strands_Trajectories.tck: replay from the dataset's directory, or copy substrate/ under $DMIPY_SIM_SURFACE_DIR. disco/ is the same walkers consolidated, and is what you want unless you are studying the fill itself.

What is certified

Layout

disco/                        THE PACK, consolidated (152.8 M rows, 187.7 GB in 237 parts): manifest.json (meta + every
                              column's byte layout), index.json (row range per voxel and pool), columns/*.safetensors
disco/reference/              pre-replayed volumes, their records, the comparison with DiSCo, tractography scores, a card
blocks/disco/block-NNNN.p1.rpk   the shards of pass 1 (+ .json summary, .run/ record); block-NNNN.rpk = a whole block
manifest.json                 the fill's recipe: substrate, grid, walk, codec, plan, variants, code commit
plan/                         walkers per voxel per pool; the 1,399 voxel blocks and their seeds
substrate/                    DiSCo's strand files, unchanged (SOURCE.md: provenance and hashes); the far field grid
certificate/disco.json        the fill's measured certificate (a scaled block, the full battery); every block inherits it
claims/, STATUS.md, smoke/, worker/   the fill's machinery: claims by file, the status page, proving runs, the worker

Contributing compute

worker/README.md: one process fills blocks in a pipeline, claims by file, no scheduler; every shard records the code commit, the host, the seed and the sha256 of what it uploaded, and carries the record of its run. Pass 1 was walked by five machines (a GH200, three L40S, a Kaggle T4) in 151 GPU-hours over two days.

Attribution

Substrate: Rafael-Patino et al. 2021 (CC BY 4.0). Replay packs: dmipy-sim (dmrai-lab), replay-pack-spec. Cite the dataset paper for the substrate and the replay paper for the packs.