Yah-daryl/cosmos_wam_cpt_161029_step_start_from_120k

Cosmos WAM CPT — step 160129 (continued from 120k)

This repository contains the Hugging Face/Safetensors export of a Cosmos3 Edge world-action model checkpoint at iteration 160129.

The run was continued from the original DCP checkpoint at iteration 120000. It was not trained from scratch. The continuation added 40129 optimization steps, ending at iteration 160129.

Continuation training changes

Compared with the preceding 120k phase, this continuation used:

• —sequence packing increased from pack32 to pack64;
• —4 nodes × 8 H100 GPUs = 32 GPUs;
• —DataLoader batch size 4 per worker, 16 workers per rank, and prefetch factor 2;
• —input resolution 256, action chunk length 16, sample stride 1, and concat_view;
• —TorchCodec video decoding with a bounded per-worker LRU decoder cache;
• —VAE encode microbatch size 32;
• —selective activation checkpointing;
• —FSDP reshard_after_forward=false;
• —model compilation enabled in the phase-2 recipe;
• —FlashAttention 3;
• —checkpoint interval reduced to 2500 steps;
• —DataLoader hard timeout 1600 seconds and NCCL heartbeat/process-group timeout 1800 seconds;
• —automatic distributed restart from the newest complete checkpoint, with up to 3 restart attempts.

The optimizer base learning rate was 2e-5. At the 120k phase boundary, the linear schedule was rebased to the existing multiplier 0.4027843298637947, giving an effective starting LR of approximately 8.0556866e-6, with no warmup and linear decay to zero over 40129 steps.

140k recovery event

Near iteration 140k, an unusually long data-loading stall left ranks out of sync and surfaced as an NCCL collective timeout. Training was restarted from the complete iter_000140000 checkpoint.

The restart restored model, optimizer, scheduler, trainer, and RNG state so optimization remained continuous. A serialized DataLoader state was not present in that checkpoint; the per-shard data cursor was reconstructed from the resumed global step.

Training data

The continuation used the processed mixed-robot training stream, including OpenNeoData ALOHA/ARX5, RDT, AgiBotWorld, InternData ALOHA, and processed RoboMIND domains. Dataset paths and raw training data are not included in this repository.

Files and scope

• —Model weights are exported as sharded Safetensors.
• —The bundled processor/tokenizer and Cosmos3-Edge vision encoder are included.
• —The exported weights are the EMA weights selected from the final training checkpoint.
• —This is an inference-oriented export. Optimizer, scheduler, trainer, RNG, and DataLoader state from the original DCP checkpoint are not included.
• —The Wan2.2 VAE is referenced through the standard Cosmos3 checkpoint registry path and is not duplicated in this repository.

Final checkpoint save time: 2026-09-23 00:24 UTC.

Base model and code

• —Base family: NVIDIA Cosmos3-Edge
• —Intended runtime: the matching Cosmos3 framework revision recorded in export_manifest.json

Users are responsible for following the applicable Cosmos3 license terms and the licenses of the training datasets.