Modelpublic

fusion-bench/Llama-3.2-1B-Instruct_Bradly-Terry-RM_Preference-700k

sourceHugging Faceupdated 2y agoView on Hugging Face

0likes19downloads

Model Card

Model Overview

RewardBench results:

Chat	Chat Hard	Safety	Reasoning
0.89	0.42	0.68	0.61

This model has been fine-tuned on the hendrydong/preference700K dataset for 2 epochs, using the Llama-3.2-1B-Instruct model as the base. See [config](./hydraoutput2024-12-0100-02-51/.hydra/config.yaml) for more details about the training hyperparameters.

[image]

Fine-tuning was done using the fusion-bench:

bash

fusion_bench --config-name llama_full_finetune \
  fabric.loggers.name=llama_full_bradley_terry_rm \
  method=lm_finetune/bradley_terry_rm \
  method.dataloader_kwargs.batch_size=8 \
  method.accumulate_grad_batches=16 \
  method.lr_scheduler.min_lr=1e-7 \
  method.lr_scheduler.max_lr=5e-6 \
  method.lr_scheduler.warmup_steps=100 \
  method.optimizer.lr=0 \
  method.optimizer.weight_decay=0.001 \
  method.gradient_clip_val=1 \
  method.max_epochs=2 \
  method.checkpoint_save_interval=epoch \
  method.checkpoint_save_frequency=1 \
  modelpool=SeqenceClassificationModelPool/llama_preference700k

8 GPUs, per-GPU batch size is 8, with gradient accumulation of 16 steps, so the effective batch size is 1024.