zhongweixie/inplace-ttt-qwen3-4b-64k-baseline

Qwen3-4B Baseline (64k, 750 steps)

This model is Qwen3-4B trained with standard training (no TTT) on 64k context length for 750 training steps. This serves as the baseline for comparison with the In-Place TTT version.

🎯 Model Overview

• —Base Model: Qwen3-4B (Qwen2.5-4B architecture)
• —Training Method: Standard (no Test-Time Training)
• —Context Length: 64k tokens
• —Training Steps: 750
• —Model Size: 8.3GB
• —Format: HuggingFace Transformers (safetensors)
• —Training Date: May 13, 2026

📊 Purpose

This is the baseline model for the In-Place TTT research project at 64k context length. It uses the same:

• —Base architecture (Qwen3-4B)
• —Training data (ProLong 64k)
• —Hyperparameters

The only difference is that it does not use In-Place Test-Time Training.

🔬 Comparison

Compare this baseline with the TTT version to see the impact of In-Place TTT:

🚀 Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Load model
model = AutoModelForCausalLM.from_pretrained(
    "zhongweixie/inplace-ttt-qwen3-4b-64k-baseline",
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)

tokenizer = AutoTokenizer.from_pretrained(
    "zhongweixie/inplace-ttt-qwen3-4b-64k-baseline",
    trust_remote_code=True
)

# Generate text with long context
prompt = "Your very long context prompt here (up to 64k tokens)..."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    temperature=0.7,
    do_sample=True
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

📈 Training Details

• —Training Data: ProLong text dataset
• —Context Length: 65,536 tokens
• —Training Steps: 750
• —Optimizer: AdamW
• —Hardware: 8x H100 GPUs

🔗 Related Models

Part of the In-Place TTT model family:

• —Qwen3-4B 64k TTT - TTT version for comparison
• —Qwen3-4B 32k TTT - 32k context version
• —Qwen3-4B 32k Baseline - 32k baseline

📦 Resources

• —GitHub Repository: https://github.com/zhongweixie/inplace_ttt
• —Results Dataset: https://huggingface.co/datasets/zhongweixie/inplace-ttt-results
• —Paper: https://arxiv.org/abs/2604.06169

📄 License

Apache 2.0 License

🎓 Citation

@inproceedings{feng2026inplace,
  title     = {In-Place Test-Time Training},
  author    = {Feng, Guhao and Luo, Shengjie and Hua, Kai and Zhang, Ge and Huang, Wenhao and He, Di and Cai, Tianle},
  booktitle = {International Conference on Learning Representations (ICLR)},
  year      = {2026},
  note      = {Oral Presentation},
  url       = {https://arxiv.org/abs/2604.06169}
}

📞 Contact

For questions or issues, please open an issue in the GitHub repository.

Model Card Author: Hansirui (zhongweixie) Uploaded: 2026-09-04 Project: In-Place Test-Time Training