respinosamena/Helios-Nova-306M-Vision

<p align="center"> <img src="heliosnovabanner.svg" alt="Helios Nova Vision" width="100%"/> </p>

Helios Nova Vision — 306M

Helios Nova Vision is a multimodal extension of the budget-efficient Helios-Nova-306M language model. By leveraging a frozen CLIP vision encoder and a lightweight, custom-trained projection layer, it grants the original text-only model the ability to "see" and describe images.

This project demonstrates how to effectively bridge continuous visual data with discrete text tokens on a strict budget. The alignment training (teaching the model its visual vocabulary) was completed on a single NVIDIA H100 in under 2 hours.

The Efficiency Story Continues

<p align="center"> <img src="datascalecomparison.svg" alt="Training data vs performance" width="100%"/> </p>

The original Helios Nova proved that competitive dense models could be trained on just 50B tokens for under $190. Helios Nova Vision applies this same philosophy to Multimodal AI.

Instead of training a Vision-Language Model (VLM) from scratch, we adopted the highly efficient "LLaVA approach." We preserved the 50B tokens of language pre-training by freezing the LLM, preserved the visual feature extraction by freezing a standard CLIP model, and focused 100% of the compute on training a 1.8M-parameter bridge between them.

Vision-Language Architecture

Helios Nova Vision consists of three distinct components working in tandem:

1. 1.The Eyes (Vision Encoder): openai/clip-vit-base-patch16. Takes an image and outputs a sequence of 197 continuous visual embeddings (1 CLS token + 196 patch tokens) at a 768-dimensional size.
2. 2.The Bridge (Projection Layer): A trainable two-layer Multi-Layer Perceptron (MLP) with a GELU activation function. This translates CLIP's 768d visual embeddings into Helios Nova's native 1024d embedding space.
3. 3.The Brain (Language Model): The frozen Helios-Nova-306M causal transformer. The translated visual tokens are prepended to the user's text tokens, allowing the model's self-attention mechanism to read the image naturally.

Why this approach?

By completely freezing the base models and only computing gradients for the 1.8M parameters in the projection layer, memory overhead is drastically reduced. This allows for massive batch sizes and lightning-fast epochs, even on a single GPU.

Training (Visual Alignment)

Phase 1: Feature Alignment

The projection layer was trained using the liuhaotian/LLaVA-CC3M-Pretrain-595K dataset. This phase acts as a "visual dictionary," teaching the model to map the visual embeddings of objects (e.g., a cat) to the specific English text tokens Helios Nova learned during its FineWeb-Edu pre-training.

Hyperparameters

AdamW optimizer · Learning rate 1e-3 · Batch size 32 · bfloat16 precision via PyTorch AMP · 1 Epoch

Limitations

• —Phase 1 Alignment Only: The model has learned basic visual vocabulary (image captioning) but has not undergone Phase 2 Visual Instruction Tuning. It cannot yet engage in complex visual Q&A (e.g., "Why is this meme funny?").
• —English Only: Inherits the English-only limitation of the base Helios Nova model.
• —Resolution Cap: Bound by CLIP's native 224x224 patch resolution. Small text in images or highly dense documents may be illegible to the model.
• —No Safety Alignment: No RLHF or safety filtering has been applied.

Talk to Helios Nova Vision

The repository includes a custom PyTorch wrapper (HeliosNovaVision.py) to handle the multimodal forward pass, as well as a command-line interface (chat_vision.py) to run inference instantly.

1. Clone the repository

git clone [https://github.com/rafaelespinosamena/Helios-Nova-306M-Vision.git](https://github.com/rafaelespinosamena/Helios-Nova-306M-Vision.git)
cd Helios-Nova-306M-Vision

2. Install dependencies

pip install torch torchvision transformers datasets Pillow huggingface_hub

3. Run Inference via CLI

The script automatically detects the best hardware (CUDA, Apple MPS, or CPU), downloads the base model, vision encoder, and projection weights, and processes your image.

python chat_vision.py --path "/path/to/your/image.jpg" --prompt "Describe this image: "

Output Example:

Device selected: mps
Loading base Helios-Nova-306M from Hub...
Downloading projection weights from Helios-Nova-306M-Vision...
Preparing inputs...

--------------------------------------------------
Helios Nova Vision: Describe this image: A small orange tabby cat sitting on a windowsill looking out at a grassy yard.
--------------------------------------------------

Reproducibility

The full training pipeline used to align the projection weights is included in train_vision.py.

GitHub: github.com/rafaelespinosamena/Helios-Nova-306M-Vision

Citation

@misc{espinosamena2026heliosvision,
  title   = {Helios Nova Vision: Multimodal Alignment of a Budget-Efficient 306M Parameter SLM},
  author  = {Espinosa Mena, Rafael},
  year    = {2026},
  url     = {[https://github.com/rafaelespinosamena/Helios-Nova-306M-Vision](https://github.com/rafaelespinosamena/Helios-Nova-306M-Vision)},
  note    = {LLaVA-style multimodal projection, LLaVA-CC3M pretraining, single H100}
}

Acknowledgements

Architecture and alignment strategy heavily inspired by LLaVA (Liu et al. 2023). Vision backbone powered by OpenAI's CLIP (Radford et al. 2021).