tussiiiii/llm-classification-v7a-ensemble-teacher-v1

llm-classification-v7a-ensemble-teacher-v1

Pseudo-label dataset for Kaggle LLM Classification Finetuning v7a.

Purpose

This dataset is generated by 01.5 ensemble pseudo-labeling.

The purpose of this dataset is to transfer the knowledge of a two-model teacher ensemble into a single student model for v7a training.

Teachers:

• —v5a: tussiiiii/llmcmp-distill-llama3-8b-lora-v5a-no-rationale-long-ab-swap-merged
• —v6q: tussiiiii/llmcmp-distill-qwen25-7b-lora-v6q-target-no-rationale-long-ab-swap-merged

Ensemble:

• —v5a weight: 0.48
• —v6q weight: 0.52

Inference settings

• —maxseqlength: 4096
• —maxprompttokens: 512
• —maxresptokens: 1536
• —userationaleinput: False
• —useabswap_tta: False
• —inferbatchsize: 2

Pseudo-label strategy

• —ensembleconfthreshold: 0.55
• —ensemblemarginthreshold: 0.08
• —trainreadymode: conservative
• —allowensembleoverride_raw: False
• —allowensembleoverride_distilled: True

The default strategy is conservative:

• —raw rows mainly use the original gold label
• —distilled rows can use the ensemble label when ensemble confidence and margin are high enough
• —the final hard label for v7a SFT is stored in target_label_for_sft

Main files

Main file for v7a SFT:

• —train_v7a_ensemble_teacher_train_ready.csv

Full diagnostic file:

• —train_v7a_ensemble_teacher_all.csv

High-confidence ensemble subset:

• —train_v7a_ensemble_teacher_high_conf.csv

Teacher disagreement subset:

• —train_v7a_ensemble_teacher_disagreements.csv

Summary metadata:

• —v7a_ensemble_teacher_summary.json

Main training column

The primary hard label column for v7a is:

• —target_label_for_sft

This column contains one of:

• —A
• —B
• —C

where:

• —A means response A is preferred
• —B means response B is preferred
• —C means tie

Useful probability columns

Ensemble probabilities:

• —ens_prob_a
• —ens_prob_b
• —ens_prob_tie
• —ens_label_abc
• —ens_conf
• —ens_margin

v5a teacher probabilities:

• —v5a_prob_a
• —v5a_prob_b
• —v5a_prob_tie
• —v5a_label_abc
• —v5a_conf
• —v5a_margin

v6q teacher probabilities:

• —v6q_prob_a
• —v6q_prob_b
• —v6q_prob_tie
• —v6q_label_abc
• —v6q_conf
• —v6q_margin

Label source columns:

• —target_source_for_sft
• —target_label_for_sft_conservative
• —target_source_for_sft_conservative
• —target_label_for_sft_aggressive
• —target_source_for_sft_aggressive

Diagnostic columns:

• —is_ens_high_conf
• —v5a_v6q_agree
• —ens_agrees_gold
• —ens_agrees_target
• —teacher_disagreement

Intended use

This dataset is intended for private Kaggle experimentation and model distillation.

The recommended student training setup is:

• —base model: Llama-3.1-8B Instruct
• —training style: winner-only SFT
• —rationale input: disabled
• —A/B swap augmentation: enabled
• —target column: target_label_for_sft

Summary

{
  "version": "v7a_ensemble_teacher_v1",
  "seed": 777,
  "teacher_configs": [
    {
      "name": "v5a_target_no_rationale_long_ab_swap",
      "model_repo_id": "tussiiiii/llmcmp-distill-llama3-8b-lora-v5a-no-rationale-long-ab-swap-merged",
      "temperature": 0.95,
      "bias": [
        0.04,
        -0.04,
        0.1
      ],
      "weight": 0.48
    },
    {
      "name": "v6q_qwen25_7b_target_no_rationale_long_ab_swap",
      "model_repo_id": "tussiiiii/llmcmp-distill-qwen25-7b-lora-v6q-target-no-rationale-long-ab-swap-merged",
      "temperature": 0.85,
      "bias": [
        -0.06,
        0.06,
        0.04
      ],
      "weight": 0.52
    }
  ],
  "max_seq_length": 4096,
  "max_prompt_tokens": 512,
  "max_resp_tokens": 1536,
  "use_rationale_input": false,
  "use_ab_swap_tta": false,
  "infer_batch_size": 2,
  "ensemble_conf_threshold": 0.55,
  "ensemble_margin_threshold": 0.08,
  "allow_ensemble_override_raw": false,
  "allow_ensemble_override_distilled": true,
  "train_ready_mode": "conservative",
  "n_all": 116112,
  "n_train_ready": 116112,
  "n_high_conf": 72342,
  "n_disagreement": 10518,
  "source_distribution_all": {
    "raw": 57477,
    "filtered": 35301,
    "safe_filtered": 22834,
    "teacher_hard": 500
  },
  "target_distribution_train_ready": {
    "A": 41690,
    "B": 40624,
    "C": 33798
  },
  "ens_label_distribution_all": {
    "A": 43787,
    "B": 42782,
    "C": 29543
  },
  "high_conf_ratio": 0.6230363786688714,
  "v5a_v6q_agree_ratio": 0.7996675623535896,
  "ens_agrees_target_ratio": 0.6696809976574342,
  "ens_agrees_gold_ratio": 0.6696809976574342,
  "target_differs_existing_ratio": 0.06816694226264297,
  "outputs": {
    "all_csv": "/content/drive/MyDrive/kaggle_cache/llmcmp_v7a_ensemble_teacher/outputs/train_v7a_ensemble_teacher_all.csv",
    "train_ready_csv": "/content/drive/MyDrive/kaggle_cache/llmcmp_v7a_ensemble_teacher/outputs/train_v7a_ensemble_teacher_train_ready.csv",
    "high_conf_csv": "/content/drive/MyDrive/kaggle_cache/llmcmp_v7a_ensemble_teacher/outputs/train_v7a_ensemble_teacher_high_conf.csv",
    "disagreements_csv": "/content/drive/MyDrive/kaggle_cache/llmcmp_v7a_ensemble_teacher/outputs/train_v7a_ensemble_teacher_disagreements.csv",
    "summary_json": "/content/drive/MyDrive/kaggle_cache/llmcmp_v7a_ensemble_teacher/outputs/v7a_ensemble_teacher_summary.json"
  }
}