Modelpublic

tussiiiii/llmcmp-distill-llama3-8b-lora-v7a-ensemble-distill-sft-conservative-adapter

sourceHugging Faceotherupdated 3mo agoView on Hugging Face

0likes3downloads

Model Card

llmcmp-distill-llama3-8b-lora-v7a-ensemble-distill-sft-conservative-adapter

LoRA adapter for Kaggle LLM Classification Finetuning.

Overview

This adapter is trained as v7a ensemble-to-single distillation.

Base Model

—unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit

Training Dataset

—Dataset: tussiiiii/llm-classification-v7a-ensemble-teacher-v1
—File: train_v7a_ensemble_teacher_train_ready.csv
—Target label column: target_label_for_sft

Training Format

—Prompt contains Prompt / Response A / Response B
—Completion is winner-only: A / B / C
—C means tie
—Rationale input is disabled
—Soft-label custom loss is not used
—A/B swap augmentation is enabled

Hyperparameters

json

{
  "model_version": "v7a-ensemble-distill-sft-conservative-seed777-lr8e6",
  "base_model": "unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit",
  "hf_v7a_dataset_id": "tussiiiii/llm-classification-v7a-ensemble-teacher-v1",
  "hf_v7a_train_file": "train_v7a_ensemble_teacher_train_ready.csv",
  "target_label_source": "target_label_for_sft",
  "seed": 777,
  "max_seq_length": 4096,
  "max_prompt_tokens": 512,
  "max_resp_tokens": 1536,
  "use_rationale_input": false,
  "include_soft_label_text": false,
  "use_ab_swap_augmentation": true,
  "final_max_rows": 0,
  "train_rows_after_processing": 124014,
  "train_rows": 122773,
  "valid_rows": 1241,
  "label_distribution_after_processing": {
    "A": 43299,
    "B": 43299,
    "C": 37416
  },
  "label_distribution_train": {
    "B": 42866,
    "A": 42866,
    "C": 37041
  },
  "label_distribution_valid": {
    "B": 433,
    "A": 433,
    "C": 375
  },
  "lora_r": 64,
  "lora_alpha": 128,
  "learning_rate": 8e-06,
  "weight_decay": 0.05,
  "per_device_train_bs": 2,
  "grad_accum": 4,
  "effective_batch_size": 8,
  "max_steps": 10000,
  "num_epochs": 1,
  "output_dir": "/content/drive/MyDrive/llmcmp_v7a/outputs_train_unsloth_v7a-ensemble-distill-sft-conservative-seed777-lr8e6",
  "adapter_dir": "/content/drive/MyDrive/llmcmp_v7a/student_adapter_v7a-ensemble-distill-sft-conservative-seed777-lr8e6",
  "merged_dir": "/content/drive/MyDrive/llmcmp_v7a/student_model_merged_v7a-ensemble-distill-sft-conservative-seed777-lr8e6",
  "hf_model_id_adapter": "tussiiiii/llmcmp-distill-llama3-8b-lora-v7a-ensemble-distill-sft-conservative-adapter",
  "hf_model_id_merged": "tussiiiii/llmcmp-distill-llama3-8b-lora-v7a-ensemble-distill-sft-conservative-merged"
}