CoolFace
Modelpublic

aplominski/TinyTransformer-Post-RMSNorm-5M-TinyStories

sourceHugging Faceopenmdw-1.1updated 26d agoView on Hugging Face
0likes
Model Card

TinyTransformer 5M - TinyStories

A 5M parameter TinyTransformer model trained on the TinyStories dataset.

This model is part of a research series investigating the effect of normalization strategies in small Transformer models. All models in the series use the same dataset and are trained under the same experimental setup, with normalization being the primary architectural variable.

Research Focus

The goal of this series is to compare different normalization strategies in small-scale Transformer architectures and evaluate their impact on training and model performance.

The series includes:

  • —Baseline - reference architecture
  • —Pre-LayerNorm - LayerNorm applied before the Transformer sublayers
  • —Post-LayerNorm - LayerNorm applied after the Transformer sublayers
  • —Pre-RMSNorm - RMSNorm applied before the Transformer sublayers
  • —Post-RMSNorm - RMSNorm applied after the Transformer sublayers

Models in This Series

ModelNormalizationDescription
TinyTransformer Baseline 5MBaselineReference model
TinyTransformer Pre-LayerNorm 5MPre-LayerNormLayerNorm before Transformer sublayers
TinyTransformer Post-LayerNorm 5MPost-LayerNormLayerNorm after Transformer sublayers
TinyTransformer Pre-RMSNorm 5MPre-RMSNormRMSNorm before Transformer sublayers
TinyTransformer Post-RMSNorm 5MPost-RMSNormRMSNorm after Transformer sublayers

Dataset

All models in this series were trained on:

TinyStories by Ronen Eldan and Yuanzhi Li

Normalization Methods

The experiments compare two commonly used normalization techniques:

Layer Normalization

Layer Normalization normalizes activations across the feature dimension and was introduced by Ba et al.

RMS Normalization

RMSNorm simplifies LayerNorm by removing the mean-centering operation and normalizing using the root mean square of the activations.

The experiments evaluate both methods in pre-normalization and post-normalization configurations.

Baseline

Just without any normalization

Citations

I'm used following papers in my reaserch:

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention Is All You Need. https://arxiv.org/abs/1706.03762
Ba, J. L., Kiros, J. R., & Hinton, G. E. (2016). Layer Normalization. https://arxiv.org/abs/1607.06450
Zhang, B., & Sennrich, R. (2019). Root Mean Square Layer Normalization. https://arxiv.org/abs/1910.07467

License

All models in this series are released under the OpenMDW-1.1 license.

For the full license text, see the OpenMDW-1.1 license.