marin-dna/marin-dna-scaling-v0.5-h896-p128M

MarinDNA v0.5 scaling ladder — 128M

This 128,484,480-parameter nucleotide-level causal language model is a member of the eight-model MarinDNA v0.5 parameter-scaling ladder developed with Marin. This repository contains only the final step-215573 checkpoint from run `dna-bolinas-scaling-v0.5-h896-p128M-43ec40`, with its tokenizer bundled. It accompanies A 1B standard Transformer rivals Evo 2 40B on variant effect prediction.

Model details

The canonical source checkpoint is gs://marin-us-east5/checkpoints/dna-bolinas-scaling-v0.5-h896-p128M-43ec40/hf/step-215573; the byte-identical evalsv2 transfer cache is `s3://oa-bolinas/snakemake/analysis/evalsv2/results/checkpoints/scaling-v0.5-h896-p128M-step-215573`. The commit-pinned training script defines the production ladder.

Loading

from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "marin-dna/marin-dna-scaling-v0.5-h896-p128M"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id)

Tokenizer and input format

The bundled tokenizer is case-insensitive and represents one nucleotide per token. Its vocabulary is [PAD]=0, [UNK]=1, [BOS]=2, a=3, c=4, g=5, t=6. Pass raw DNA strings containing A, C, G, and T without spaces. The tokenizer lowercases input and prepends [BOS]; it has no EOS token. Other symbols map to [UNK]. Because BOS occupies one of the 256 positions, inputs are limited to 255 DNA bases.

Training data and protocol

Every model in the ladder used the same batch size, token budget, optimizer hyperparameters, tokenizer, and three-way training mixture: 73.19% CDS, 20.62% upstream, and 6.19% downstream sequence. Lowercase soft-masked positions received loss weight 0.01, versus 1.0 for uppercase positions. The ladder changes model scale, not the training-data recipe.

Training datasets: CDS, upstream, and downstream. The matched CDS, upstream, and downstream datasets were validation probes, not training data. The exact tokenizer, data, and mixture definitions are commit-pinned here.