distily/distily_norm_distilgpt2_sweep_extended

Summary

Distilled with Distily library using teacher model gpt2 on dataset wikimedia/wikipedia.

<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment.

Model description

More information needed

Intended uses & limitations

More information needed -->

Model Architecture:

• —Architecture: GPT2LMHeadModel
• —Total Parameters: 81,912,576
• —Data Type (dtype): torch.bfloat16
• —Model Size: 0.16 GB

Benchmark Metrics Comparison

Resource Usage Comparison

• —VRAM Use: 15.6991 GB

Distillation (Teacher -> Student) Architecture Difference:

• —Architecture: GPT2LMHeadModel -> GPT2LMHeadModel
• —Total Parameters: 124,439,808 -> 81,912,576
• —Data Type (dtype): torch.bfloat16 -> torch.bfloat16
• —Model Size: 0.24 GB -> 0.16 GB

<details> <summary>Module Diff Details</summary>

--- teacher model modules
+++ student model modules
@@ -4,7 +4,7 @@
     (wpe): Embedding(1024, 768)
     (drop): Dropout(p=0.1, inplace=False)
     (h): ModuleList(
-      (0-11): 12 x GPT2Block(
+      (0-5): 6 x GPT2Block(
         (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
         (attn): GPT2FlashAttention2(
           (c_attn): Conv1D()

</details> <br/>

Train Dataset

Trained on 521,413,804 tokens from the wikimedia/wikipedia dataset.

• —Num Samples: 990,000
• —Subset: 20231101.en
• —Split: train

Training Objective

DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl), attn_loss_component=LossComponent(label=attn, weight=5, loss_fn=raw_mse, layer_mapper=layer-2, norm=instance_teacher_only, projector=mlp))

Hyperparameters

The following hyperparameters were used during training:

<details> <summary>Expand</summary>

• —learning_rate: 0.0002
• —trainbatchsize: 8
• —evalbatchsize: 8
• —seed: 42
• —optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• —lrschedulertype: polynomial
• —num_epochs: 1.0
• —distillationobjective: `DistillationObjective(logitslosscomponent=LossComponent(label=logits, weight=1, lossfn=kl), attnlosscomponent=LossComponent(label=attn, weight=5, lossfn=rawmse, layermapper=layer-2, norm=instanceteacher_only, projector=mlp))`
• —train_embeddings: True
• —lrscheduler: `<torch.optim.lrscheduler.LambdaLR object at 0x7f99b4ed13c0>`
• —studentmodelnameorpath: None
• —studentconfignameorpath: distilbert/distilgpt2
• —studentmodelconfig: None
• —reinitialize_weights: None
• —copyteachermodules: [('lm_head', False)]
• —studentmodelas_bitnet: False
• —dropout: None
• —teachermodelnameorpath: gpt2
• —teacherloadin_8bit: False
• —teacherloadin_4bit: False
• —dataset_uri: wikimedia/wikipedia
• —dataset_subset: 20231101.en
• —dataset_split: train
• —datasetcolumnname: text
• —datasetsamplesize: 1000000
• —datasettestsize: 0.01
• —gradientaccumulationsteps: 1
• —weight_decay: 0.0
• —maxgradnorm: 1.0
• —warmup_ratio: 0
• —warmup_steps: 0
• —gradient_checkpointing: True

</details> <br/>

Framework Versions

• —Distily 0.4.1
• —Transformers 4.44.2
• —Pytorch 2.4.0+cu121
• —Datasets 2.21.0