bishmoy/Llama-3.1-8B-ReLU

Llama-3.1-8B-ReLU

This repository contains a ReLU-fied variant of Llama 3.1 8B released as part of the ReLU-Tune project.

The model was created by replacing the transformer MLP activation with ReLU and then recovering performance through staged LoRA fine-tuning. The weights in this repository are provided as a merged checkpoint for straightforward loading and evaluation.

What This Model Is

• —Base model: meta-llama/Llama-3.1-8B
• —Architecture change: transformer MLP activations changed to ReLU
• —Training method: staged LoRA fine-tuning
• —Model format: merged dense model
• —Extra metadata: activation_config.json documents the activation modification

Intended Use

This model is intended for:

• —research on activation functions in large language models
• —experiments on sparse activations and efficiency tradeoffs
• —evaluation and fine-tuning workflows built on top of ReLU-Tune

This is primarily a research release, not a production-optimized instruction model.

Files

Typical files in this repository include:

• —model.safetensors
• —config.json
• —generation_config.json
• —tokenizer.json
• —tokenizer_config.json
• —activation_config.json

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "bishmoy/Llama-3.1-8B-ReLU"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
)

Notes

• —activation_config.json is included for reproducibility and to document how activations were modified.
• —This release follows the base Llama 3.1 licensing and usage constraints. Users should review and comply with the original model terms.

Project

This model was produced with **ReLU-Tune**, a toolkit for:

• —full or partial ReLU-fication
• —staged LoRA fine-tuning
• —benchmark and perplexity evaluation
• —activation sparsity measurement

Limitations

This is an experimental model release. ReLU-fication changes the base architecture and can alter behavior in ways that differ from the original dense model. Downstream performance and robustness should be validated for any intended use.