Modelpublic

kdhole/Llama-3.1-8B-tuned-lens

sourceHugging Faceopenrailupdated 2y agoView on Hugging Face

0likes13downloads

Model Card

This is the tuned lens version of LLama 3.1 8B (i.e. all the layers of the model have been trained to minimize the KL divergence with the last layer).

bash

pip install tuned-lens

bash

python -m tuned_lens train \
  --model.name meta-llama/Llama-3.1-8B \
  --data.name bookcorpus/bookcorpus \
  --per_gpu_batch_size=1 \
  --output my_lenses/meta-llama/Llama-3.1-8B

bash

python -m tuned_lens eval \
  --data.name bookcorpus/bookcorpus \
  --model.name meta-llama/Llama-3.1-8B \
  --tokens 16400000 \
  --lens_name my_lenses/meta-llama/Llama-3.1-8B \
  --output evaluation/meta-llama/Llama-3.1-8B