dogtooth/open-lm-3b-202101

Open LM 3B — Knowledge Cutoff January 2021

This is a HuggingFace-format conversion of the Apple Open LM 3B oracle model trained with a knowledge cutoff of January 2021, from the TiC-LM (Time-Continual Language Modeling) project.

Model Details

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "dogtooth/open-lm-3b-202101",
    dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-neox-20b")

Conversion Notes

• —Converted from the original Open LM .pt checkpoint to a custom OpenLMForCausalLM format.
• —Uses LayerNorm (not RMSNorm) to match the original Open LM training.
• —Includes QK norm (LayerNorm on Q and K projections before attention).
• —Architecture dimensions are auto-detected from checkpoint weights.
• —Requires trust_remote_code=True when loading.

Citation

@article{jain2024ticlm,
  title={Time-Continual Learning from a Streaming Language Model},
  author={Jain, Ameya and Ramesh, Aakanksha and Li, Tianjian and others},
  journal={arXiv preprint arXiv:2410.14660},
  year={2024}
}