Paranioar/NEO1_0-9B-SFT

<div align='center'> <h1>From Pixels to Words -- Towards Native Vision-Language Primitives at Scale</h1h1>

| Paper | Code | </div>

🌟🌟 Motivation

Two lingering clouds cast shadows over its widespread exploration and promotion:

• —What fundamental constraints set native VLMs apart from modular ones, and to what extent can these barriers be overcome?

• —How to make research in native VLMs more accessible and democratized, thereby accelerating progress in the field.

We construct native VLMs built from first principles, where its primitive should:

• —effectively align pixel and word representations within a shared semantic space;

• —seamlessly integrate the strengths of separate vision and language modules;

• —inherently embody various cross-modal properties that support unified vision-language encoding, aligning, and reasoning.

🚀🚀 Highlight

• —With only 390M image-text examples, NEO develops strong visual perception from scratch inside a dense and monolithic model via elaborate primitives.

• —NEO serves as a cornerstone for scalable and powerful native VLMs, paired with reusable components that foster a cost-effective and extensible ecosystem.

🧑‍🎨🧑‍🎨 Model Overview

NEO1_0-9B has the following features:

• —Model Type: Native Vision-Language Models

• —Model Mode: Mixed Native-Attn & Native-RoPE

• —Layer Parameters: 214M vs. 193M (Qwen3-8B)

• —Model Parameters: 9B (Non-Embedding)

• —Number of Layers: 42 (6 for Pre-Buffer & 36 for Post-LLM)

• —Number of Heads: 32 for Q and 8 for KV (GQA)

• —Head Dimensions: 128 * 2 for QK and 128 for V

🔥🔥 Model Performance

<img src="https://cdn-uploads.huggingface.co/production/uploads/64b4a717aa03b6520839e9b8/9V6RoCK2EmdLtDe6Jw04m.png" width="600">

<img src="https://cdn-uploads.huggingface.co/production/uploads/64b4a717aa03b6520839e9b8/pO5rbMpKomePyfbk90LrQ.png" width="600">

📚📚 Model Weights

We release the 9B weights of NEO1_0 in Pre-Training (PT), Mid-Training (MT), and Supervised Fine-Tuning (SFT).

✒️✒️ Citation

If NEO is helpful for your research, please consider star ⭐ and citation 📝 :

@article{Diao2025NEO,
  title        = {From Pixels to Words--Towards Native Vision-Language Primitives at Scale},
  author       = {Diao, Haiwen and Li, Mingxuan and Wu, Silei and Dai, Linjun and Wang, Xiaohua and Deng, Hanming and Lu, Lewei and Lin, Dahua and Liu, Ziwei},
  journal      = {arXiv preprint arXiv:2510.14979},
  year         = {2025}
}