ArchSpace-Collection/NCP_ArchPreview_dolma3_8.9B_Stage1_Step700000

NCP-ArchPreview 8.9B - Stage 1 - Step 700,000

Model collection | Technical report (arXiv) | HF Papers | Training code (coming soon) | Evaluation code

NCP-ArchPreview is a latent-space autoregressive language model developed by The NCP Team at Shanghai AI Lab and LUMIA Lab, Shanghai Jiao Tong University. It learns to predict both the next token and the next concept: a representation spanning a short group of tokens in a learned latent space. Concept predictions guide the token decoder, while generation retains the standard next-token interface.

This repository contains the intermediate Stage 1 checkpoint at step 700,000, saved during pretraining on Dolma 3 Mix. The architecture follows the OLMo 3 7B token-level design and adds a Concept Module, a product-quantized concept vocabulary, and hierarchical residual connections, bringing the total parameter count to approximately 8.94B.

Highlights

• —Joint token and concept learning. Next Concept Prediction (NCP) supplies explicit supervision over a latent sequence at one quarter of the token sequence length.
• —Pretraining at scale. The report describes training on 5.73T Dolma-3 tokens. Stage 1 reaches the OLMo-3-7B final training loss using 51.3% of its training tokens, corresponding to 1.95x convergence in token budget.
• —Stronger Stage 1 results. The report's main evaluation gives an Overall AVG of 49.04, compared with 46.59 for OLMo-3-7B, including +5.99 percentage points on GSM8K and +4.28 points on HumanEval.
• —A learned interface for adaptation. Separate experiments adapt the existing concept codebooks and prediction heads, approximately 17M parameters, while keeping the token backbone fixed.

The convergence comparison measures tokens required to reach a reference loss; it does not measure wall-clock training speed or inference throughput.

Architecture

NCP-ArchPreview processes text through three modules:

1. 1.A 16-layer Token Encoder produces contextual token states. Mean pooling over each group of four states forms a continuous concept representation.
2. 2.An 8-layer Concept Module predicts the next concept. Product quantization defines the concept vocabulary using 32 codebooks. Predictions are differentiable weighted combinations of codewords.
3. 3.A 16-layer Token Decoder receives token states and causally aligned concept predictions, then produces the next-token distribution.

Intra-module residual connections mix states across depths. Cross-module residual connections connect Encoder to Concept Module, Encoder to Decoder, and Concept Module to Decoder. Concept feedback is shifted and repeated at token resolution to preserve causality.

tokens -> Token Encoder -> mean pooling -> Concept Module -> concept prediction
                 |                                                |
                 +--------------> Token Decoder <-----------------+
                                       |
                                next-token logits

Training

Stage 1 uses Dolma 3 Mix and the staged pretraining framework described in the report. The reported 5.73T-token budget describes the large-scale training run; intermediate checkpoints have consumed only the tokens preceding their saved step.

The model is optimized jointly with three objectives:

• —NTP: the standard causal next-token cross-entropy loss.
• —NCP: prediction of the next continuous concept through the learned codebooks, with a stop-gradient target.
• —VQ: fitting codebook entries to the encoder's concept representations.

The report uses Moonlight Muon for matrix-valued parameters and AdamW for embeddings, biases, and other non-Muon parameters. Its default learning rate is 6e-5, with the OLMo-3-style cosine schedule.

Evaluation

Results below describe the report's final Stage 1 model and are included as reference results for this intermediate checkpoint. Scores are percentages and higher is better; deltas are absolute percentage points.

<details> <summary>Results by domain and likelihood evaluation</summary>

Likelihood is reported separately in bits per UTF-8 byte (BPB), where lower is better.

The complete per-benchmark results are available in Table 1.

</details>

Overall AVG is the unweighted mean of the 26 constituent benchmark scores in Table 1, excluding the aggregate MMLU row, domain averages, and BPB results. BPB AVG is computed separately over ten likelihood benchmarks.

Detailed evaluation settings can be found in our technical report.

Checkpoint Trajectory

The final Stage 1 model is available as `ArchSpace-Collection/NCP_ArchPreview_dolma3_8.9B_Stage1`. The report also evaluates intermediate checkpoints from 100K to 1.3M training steps. Selected measurements from different training steps are shown below.

This repository contains checkpoint 700,000. The trajectory table lists selected training steps; use its matching row when present. The final checkpoint's scores are reference results for this checkpoint. Intermediate checkpoints support studies of training dynamics, and individual benchmark scores can fluctuate between checkpoints.

Quick Start

The checkpoint includes custom Transformers model code. This example uses a single prompt on one CUDA GPU with sufficient memory for the BF16 weights, activations, and cache.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "ArchSpace-Collection/NCP_ArchPreview_dolma3_8.9B_Stage1_Step700000"
device = "cuda"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
).eval().to(device)

prompt = "The role of hierarchical representations in language modeling is"
inputs = tokenizer(prompt, return_tensors="pt").to(device)

with torch.inference_mode():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=False,
        use_cache=True,
        pad_token_id=tokenizer.eos_token_id,
    )

continuation = output_ids[0, inputs["input_ids"].shape[1]:]
print(tokenizer.decode(continuation, skip_special_tokens=True))

Use plain completion or few-shot prompts for this base model. The example is a loading and generation example; benchmark reproduction requires the prompts, sampling configuration, and scorers described in the report. For reproducible runs, pin the model and tokenizer to the same Hub commit with revision. See the inference guide for supported runtime versions and optimized serving.

Intended Use and Limitations

This release supports research on latent-space language modeling, evaluation of pretrained capabilities, analysis of training dynamics, continued pretraining, and concept-based adaptation.

• —It is a pretrained base model and has not been aligned as a conversational assistant. It can produce inaccurate, biased, or harmful content.
• —The learned concepts are codebook-based latent representations; individual codes are not guaranteed to correspond to human-interpretable concepts.
• —The reported training context is 8,192 tokens. This card makes no claim of validated capability beyond that length.
• —Results depend on the checkpoint, prompting, sampling, and evaluation implementation. The separate adaptation and recipe-screening experiments use their own reported evaluation results.

Citation

@misc{ncpteam2026ncparchpreviewtechnicalreportmoving,
      title={NCP-ArchPreview Technical Report: Moving towards Latent Space Language Models through Next Concept Prediction}, 
      author={NCP Team and Jiaqi Cao and Chiyu Chen and Shuang Cheng and Xu Cheng and Beiya Dai and Yufan Feng and Kewen Ge and Ruijun Ge and Jiayi Huang and Yang Jiao and Dahua Lin and Zhouhan Lin and Yifan Liu and Yuliang Liu and Biqing Qi and Mowen Ruan and Junzhe Shen and Yunchong Song and Hao Sun and Zhongbo Tian and Yixuan Wang and Rubin Wei and Jiaxin Xiong and Kangyu Yang and Qian Yao and Qi Zhang and Bowen Zhou},
      year={2026},
      eprint={2609.10715},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2609.10715}, 
}

License

The model weights are released under the Apache License 2.0.

Acknowledgements

We thank the OLMo and Dolma teams and the contributors to the open datasets, training libraries, and evaluation tools used in this work.

<!-- Editorial source: ConceptOlmo7B.pdf, report dated 2026-09-04, SHA-256 acf619d01c0ad46e4012b9dacf1af7c2b21501a510008e6dad57b035fd1eaf9e. Architecture: Sections 2-3 and Table 10. Main evaluation: Table 1. Checkpoint trajectory: Table 14. Evaluation protocol: Appendix E / Table 13. Keep these tables separate from the frozen results in Section 5.1 / Appendix C. -->