Modelpublic

BAJUKA/LLaVA-llama32-3b-VLfull-s42-s2b

sourceHugging Facellama3.2updated 27d agoView on Hugging Face

0likes27downloads

Model Card

LLaVA-llama32-3b-VLfull-s42-s2b — VL-full

VL-full arm of a controlled VL-vs-text training grid built on LLaVA-NeXT. Arm definition: S1 -> S2a -> S2b. LLaVA-OneVision three-stage analog.

Backbone llama32-3b (meta-llama/Llama-3.2-3B-Instruct), seed 42, output of S2b (instruct stage).

This is a vision-language arm: it trains the full VL stack on top of the stage-1 projector, with images at anyres_max_9 through google/siglip-so400m-patch14-384.

What the grid controls

Every arm sees the same mixtures, the same example order for a given seed, the same optimizer settings and the same restarted LR schedule. Arms differ only in data, order, and which modules are trainable — so differences between arms are attributable, not confounded.

Because VLfull runs its stages sequentially with the seed inherited, the S2a checkpoint is the cap-only arm at that seed. Both stages are published separately for exactly that reason.

Training


Stage	S2b (instruct stage)
Initialized from	the S2a checkpoint of this arm (`LLaVA-llama32-3b-VLfull-s42/s2a`)
Data mixture	INS-750K — instruct700kv2 (697,850) + language50kv2 (49,956)
Trainable	projector + vision tower + LM (`mm_vision_tower,mm_mlp_adapter,mm_language_model`)
Steps	5842 / 5842 (epoch 1.0000 of 1)
Global batch	128
LR	1e-5 (LM / projector), 2e-06 (vision tower), cosine, warmup ratio 0.03
Precision	bfloat16
Max seq length	8192
Hardware	4x H100 80GB (DeepSpeed ZeRO-3)

Training loss 0.935 -> 0.525 (mean of last 50 logged steps: 0.568), 0 non-finite losses across 5842 logged steps. The run reached a full epoch; trainer_state.json is included in this repo with the complete per-step loss / grad-norm / LR history.

Usage

These weights use the LlavaLlamaForCausalLM class from the LLaVA-NeXT repo, which is not part of transformers. Load them with that repo checked out:

python

from llava.model.builder import load_pretrained_model

tokenizer, model, image_processor, ctx_len = load_pretrained_model(
    "BAJUKA/LLaVA-llama32-3b-VLfull-s42-s2b", None, "llava_llama"
)

Prompt template: llama_v3.

Caveats

—Research artifact from a controlled comparison, not a tuned or safety-aligned release. It inherits the licence and the limitations of meta-llama/Llama-3.2-3B-Instruct.
—The intermediate (s2b) checkpoint is a legitimate arm of the grid, not a "best" checkpoint — no early stopping or checkpoint selection was performed. Each stage simply runs one full epoch.
—Optimizer / DeepSpeed / RNG state is not published; these are inference weights.