timinar/baby-llama-58m

Baby Llama

Our submission to the strict-small track of the BabyLM challenge.

Baby Llama is a 58M-parameter model, distilled from an ensemble consisting of LLaMA-360M and GPT2-705M, both trained on the babylm_10M dataset.

See the associated paper for a detailed discussion of the training procedure and of the model performance. The training code is available at https://github.com/timinar/BabyLlama.

Hyperparameters for the tasks that require fine-tuning

When evaluating the model on the tasks that require fine-tuning, we noticed that the default hyperparameters suggested by the BabyLM organizers lead to severe overfitting in a number of tasks. To avoid this issue, we have re-tuned those hyperparameters. The sets of hyperparameters selected for each task are listed in the table below.