Rubin-Wei/MemoryDecoder-Pythia-6.9B-general

1---2library_name: transformers3license: apache-2.04tags:5- memory-decoder6- parametric-memory7- long-term-memory8- memorydecoder-at-scale9- text-generation10- pythia11---12# MemoryDecoder-Pythia-6.9B-general13 14## Resources15 16- [Project Page: Memory Decoder at Scale](https://rubin-wei.github.io/memory-decoder-at-scale/)17- [GitHub Repository: LUMIA-Group/MemoryDecoder-at-Scale](https://github.com/LUMIA-Group/MemoryDecoder-at-Scale)18- [Paper: Memory Decoder at Scale: A Pretrained, Parametric Long-Term Memory](https://arxiv.org/abs/2607.27919)19- [Hugging Face Collection: MemoryDecoder-at-Scale](https://huggingface.co/collections/Rubin-Wei/memorydecoder-at-scale)20 21This repository contains the **6.9B general Memory Decoder** released with22[Memory Decoder at Scale](https://arxiv.org/abs/2607.27919). It is a pretrained parametric long-term23memory designed to be paired with a frozen Pythia-family language model. The24general-memory suite was pretrained on 300B tokens and scales memory capacity25independently from the backbone.26 27This checkpoint is a memory component, not a chat- or instruction-tuned model.28Use it through the Memory Decoder integration in the released codebase rather29than treating it as a standalone assistant.30 31## Model details32 33| Field | Value |34|---|---|35| Memory size | approximately 6.9B parameters |36| Architecture/tokenizer family | GPT-NeoX / Pythia |37| Memory scope | General |38| Training scale | 300B-token general-memory pretraining suite |39| Intended backbone | Frozen, tokenizer-compatible Pythia model |40| Release contents | Inference weights, configuration, and tokenizer files |41 42## Usage43 44Install the environments from45[LUMIA-Group/MemoryDecoder-at-Scale](https://github.com/LUMIA-Group/MemoryDecoder-at-Scale), then use the46`hf-memdec` adapter. For example:47 48```bash49cd eval/lm-evaluation-harness50 51BACKBONE=EleutherAI/pythia-410m-deduped52MEMORY=Rubin-Wei/MemoryDecoder-Pythia-6.9B-general53 54lm-eval \55  --model hf-memdec \56  --model_args pretrained=$BACKBONE,memdec_path=$MEMORY \57  --tasks arc_easy,piqa,mmlu \58  --batch_size 159```60 61The adapter provides the task-specific interpolation settings used for the62released Pythia memories; an explicit `lmbda` can be supplied as an override.63 64## Intended use and limitations65 66This checkpoint is intended for research on parametric memory, memory scaling,67and evaluation with frozen language-model backbones. Its behavior depends on68the selected backbone and interpolation settings. It may reproduce biases or69errors present in its training data and should not be treated as an authoritative70knowledge source.71 72## Citation73 74If you use this checkpoint, please cite:75 76```bibtex77@misc{wei2026memorydecoderscalepretrained,78      title={Memory Decoder at Scale: A Pretrained, Parametric Long-Term Memory},79      author={Rubin Wei and Jiaqi Cao and Jiarui Wang and Junming Zhang and Qipeng Guo and Bowen Zhou and Zhouhan Lin},80      year={2026},81      eprint={2607.27919},82      archivePrefix={arXiv},83      primaryClass={cs.CL},84      url={https://arxiv.org/abs/2607.27919},85}86```87