Saiteja23/smollm2-135m-fineweb-edu2.7BT

Saiteja23/smollm2-135m-fineweb-edu2.7BT

Model Description

This is a 135M parameter language model based on the SmolLM2 architecture, trained on FineWeb-edu data.

Model Details

• —Model Type: Causal Language Model (Decoder-only Transformer)
• —Architecture: LLaMA-style with Grouped Query Attention
• —Parameters: ~135M
• —Training Data: 2.7B tokens from FineWeb-edu
• —Base Model: HuggingFaceTB/SmolLM2-135M (tokenizer)
• —License: Apache 2.0

Architecture Details

• —Hidden Size: 576
• —Attention Heads: 9
• —Key-Value Heads: 3 (Grouped Query Attention)
• —Layers: 30
• —Vocabulary Size: 49,152
• —Max Position Embeddings: 8,192
• —Max Sequence Length: 512
• —RoPE Theta: 100,000

Training Details

• —Training Regime: Pretraining (no instruction tuning)
• —Dataset: FineWeb-edu (sample-100BT)
• —Training Tokens: 2.7B tokens
• —Sequence Length: 512
• —Batch Size: 128
• —Learning Rate: 1e-3
• —Optimizer: AdamW
• —Precision: bfloat16

Usage

Text Generation

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

# Load model and tokenizer
tokenizer = AutoTokenizer.from_pretrained("Saiteja23/smollm2-135m-fineweb-edu2.7BT")
model = AutoModelForCausalLM.from_pretrained("Saiteja23/smollm2-135m-fineweb-edu2.7BT")

# Generate text
prompt = "The process of photosynthesis"
inputs = tokenizer(prompt, return_tensors="pt")

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=100,
        temperature=0.8,
        top_k=50,
        do_sample=True,
        pad_token_id=tokenizer.eos_token_id
    )

generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(generated_text)

Custom Generation (using original model code)

# If you want to use the original model implementation
from model import ModelConfig, LlamaModel
from transformers import AutoTokenizer
import torch

tokenizer = AutoTokenizer.from_pretrained("Saiteja23/smollm2-135m-fineweb-edu2.7BT")
# Load using custom model class - see repository for details

Model Performance

This model is designed for:

• —Text Continuation: Completing partial sentences or paragraphs
• —Educational Content: Generating educational explanations and content
• —Creative Writing: Short-form creative text generation
• —General Language Modeling: Basic language understanding and generation

Limitations

• —No Instruction Following: This is a pretrained model without instruction tuning
• —No Safety Training: No specific safety or alignment training applied
• —Educational Bias: Training data bias toward educational content
• —Size Limitations: 135M parameters limit complex reasoning capabilities

Training Infrastructure

• —Hardware: Single A100 80GB GPU
• —Framework: PyTorch with custom training loop
• —Distributed Training: Single GPU
• —Mixed Precision: bfloat16

Citation

If you use this model, please cite:

@misc{smollm2-135m-fineweb-edu,
  title={SmolLM2-135M trained on FineWeb-edu},
  author={Your Name},
  year={2024},
  howpublished={\url{https://huggingface.co/Saiteja23/smollm2-135m-fineweb-edu2.7BT}},
}

Acknowledgments

• —Base architecture inspired by SmolLM2 (HuggingFace)
• —Training data from FineWeb-edu dataset
• —Built with PyTorch and Hugging Face Transformers

This model was trained as part of a language model pretraining experiment. Use responsibly and be aware of potential biases in generated content.