Modelpublic

distily/distily_smollm_dataset_sweep

sourceHugging Facecreativeml-openrail-mupdated 2y agoView on Hugging Face

0likes24downloads

Model Card

Summary

Distilled with Distily library using teacher model HuggingFaceTB/SmolLM-135M on dataset wikimedia/wikipedia.

<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment.

Model description

More information needed

Intended uses & limitations

More information needed -->

Model Architecture:

—Architecture: LlamaForCausalLM
—Total Parameters: 81,413,568
—Data Type (dtype): torch.float32
—Model Size: 0.30 GB

<details> <summary>Student Model Details</summary>

LlamaForCausalLM(
  (model): LlamaModel(
    (embed_tokens): Embedding(49152, 576)
    (layers): ModuleList(
      (0-14): 15 x LlamaDecoderLayer(
        (self_attn): LlamaSdpaAttention(
          (q_proj): Linear(in_features=576, out_features=576, bias=False)
          (k_proj): Linear(in_features=576, out_features=192, bias=False)
          (v_proj): Linear(in_features=576, out_features=192, bias=False)
          (o_proj): Linear(in_features=576, out_features=576, bias=False)
          (rotary_emb): LlamaRotaryEmbedding()
        )
        (mlp): LigerSwiGLUMLP(
          (gate_proj): Linear(in_features=576, out_features=1536, bias=False)
          (up_proj): Linear(in_features=576, out_features=1536, bias=False)
          (down_proj): Linear(in_features=1536, out_features=576, bias=False)
        )
        (input_layernorm): LigerRMSNorm((576,), eps=1e-05, offset=0.0)
        (post_attention_layernorm): LigerRMSNorm((576,), eps=1e-05, offset=0.0)
      )
    )
    (norm): LigerRMSNorm((576,), eps=1e-05, offset=0.0)
    (rotary_emb): LlamaRotaryEmbedding()
  )
  (lm_head): Linear(in_features=576, out_features=49152, bias=False)
)

</details> <br/>

Benchmark Metrics Comparison

—student 0: dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=20231101.en, dataset_uri=wikimedia_wikipedia, per_device_train_batch_size=8
—student 1: dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=None, dataset_uri=distily_filtered_redpajama_en, per_device_train_batch_size=8
—student 2: dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=sample-10BT, dataset_uri=HuggingFaceFW_fineweb-edu, per_device_train_batch_size=8
—student 3: dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=sample-10BT, dataset_uri=HuggingFaceFW_fineweb, per_device_train_batch_size=8
—student 4: dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=sample-10BT, dataset_uri=HuggingFaceFW_fineweb, learning_rate=6e-05, per_device_train_batch_size=8
—student 5: dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=sample-10BT, dataset_uri=HuggingFaceFW_fineweb-edu, learning_rate=6e-05, per_device_train_batch_size=8
—student 6: dataset_max_seq_length=1024, dataset_sample_size=4000000, dataset_subset=20231101.en, dataset_uri=wikimedia_wikipedia, per_device_train_batch_size=8
—student 7: dataset_max_seq_length=1024, dataset_sample_size=4000000, dataset_subset=20231101.en, dataset_uri=wikimedia_wikipedia, learning_rate=6e-05, per_device_train_batch_size=8
—student 8: dataset_max_seq_length=1024, dataset_sample_size=4000000, dataset_shuffle=True, dataset_subset=20231101.en, dataset_uri=wikimedia_wikipedia, per_device_train_batch_size=8

Metric	teacher	student 0	student 1	student 2	student 3	student 4	student 5	student 6	student 7	student 8
tinyArc.acc_norm,none	0.37	0.303	0.295	0.302	0.26	0.269	0.319	0.286	0.299	0.316
tinyGSM8k.exact_match,flexible-extract	0.006	0.029	0.03	0.025	0.006	0.006	0.012	0.012	0.017	0.006
tinyGSM8k.exact_match,strict-match	0.006	0.006	0.006	0.006	0.006	0.006	0.006	0.006	0.006	0.006
tinyHellaswag.acc_norm,none	0.452	0.341	0.281	0.327	0.3	0.303	0.301	0.364	0.356	0.348
tinyMMLU.acc_norm,none	0.341	0.276	0.281	0.31	0.286	0.279	0.292	0.295	0.328	0.311
tinyTruthfulQA.acc,none	0.38	0.463	0.447	0.423	0.419	0.421	0.427	0.44	0.436	0.433
tinyWinogrande.acc_norm,none	0.509	0.466	0.436	0.46	0.492	0.473	0.417	0.439	0.482	0.503

Resource Usage

—Max Train VRAM Use: 13.1269 GB
—Available VRAM: 23.4329 GB
—GPUs:
—1x NVIDIA GeForce RTX 4090
—CPUs: 64
—CPU Memory: 251.7299 GB
—CPU Memory Bandwidth: 1600 GB/s

Distillation (Teacher -> Student) Architecture Difference:

—Architecture: LlamaForCausalLM -> LlamaForCausalLM
—Total Parameters: 134,515,008 -> 81,413,568
—Data Type (dtype): torch.float32 -> torch.float32
—Model Size: 0.25 GB -> 0.30 GB

<details> <summary>Module Diff Details</summary>

diff

--- teacher model modules
+++ student model modules
@@ -2,7 +2,7 @@
   (model): LlamaModel(
     (embed_tokens): Embedding(49152, 576)
     (layers): ModuleList(
-      (0-29): 30 x LlamaDecoderLayer(
+      (0-14): 15 x LlamaDecoderLayer(
         (self_attn): LlamaSdpaAttention(
           (q_proj): Linear(in_features=576, out_features=576, bias=False)
           (k_proj): Linear(in_features=576, out_features=192, bias=False)
@@ -10,17 +10,16 @@
           (o_proj): Linear(in_features=576, out_features=576, bias=False)
           (rotary_emb): LlamaRotaryEmbedding()
         )
-        (mlp): LlamaMLP(
+        (mlp): LigerSwiGLUMLP(
           (gate_proj): Linear(in_features=576, out_features=1536, bias=False)
           (up_proj): Linear(in_features=576, out_features=1536, bias=False)
           (down_proj): Linear(in_features=1536, out_features=576, bias=False)
-          (act_fn): SiLU()
         )
-        (input_layernorm): LlamaRMSNorm((576,), eps=1e-05)
-        (post_attention_layernorm): LlamaRMSNorm((576,), eps=1e-05)
+        (input_layernorm): LigerRMSNorm((576,), eps=1e-05, offset=0.0)
+        (post_attention_layernorm): LigerRMSNorm((576,), eps=1e-05, offset=0.0)
       )
     )
-    (norm): LlamaRMSNorm((576,), eps=1e-05)
+    (norm): LigerRMSNorm((576,), eps=1e-05, offset=0.0)
     (rotary_emb): LlamaRotaryEmbedding()
   )
   (lm_head): Linear(in_features=576, out_features=49152, bias=False)

</details> <br/>

Train Dataset

Trained on 1,911,742,377 tokens from the wikimedia/wikipedia dataset.

—Num Samples: 3,992,000
—Subset: 20231101.en
—Split: train

Training Objective

DistillationObjective(
    logits_loss_component=LossComponent(
        weight=1,
        loss_fn='kl'
    ),
    hs_loss_component=LossComponent(
        weight=0
    ),
    attn_loss_component=LossComponent(
        weight=0
    )
)

Hyperparameters

The following hyperparameters were used during training:

<details> <summary>Expand</summary>

—learning_rate: 0.0001
—trainbatchsize: 8
—evalbatchsize: 4
—seed: 42
—optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
—lrschedulertype: polynomial
—lrschedulerwarmup_ratio: 0.1
—num_epochs: 1.0
—distillationobjective: `DistillationObjective( logitslosscomponent=LossComponent( weight=1, lossfn='kl' ), hslosscomponent=LossComponent( weight=0 ), attnlosscomponent=LossComponent( weight=0 ) )`
—lrscheduler: `<torch.optim.lrscheduler.LambdaLR object at 0x7f3e58c6d840>`
—studentmodelnameorpath: None
—studentconfignameorpath: None
—studentmodelconfig: {'num_hidden_layers': 15}
—reinitialize_weights: None
—copyteachermodules: [('lm_head', False)]
—studentmodelas_bitnet: False
—studentuseliger_kernel: True
—teachermodelnameorpath: HuggingFaceTB/SmolLM-135M
—teacherloadin_8bit: False
—teacherloadin_4bit: False
—dataset_uri: wikimedia/wikipedia
—dataset_subset: 20231101.en
—dataset_split: train
—datasetcolumnname: text
—datasetsamplesize: 4000000
—datasetmaxseq_length: 1024
—datasettestsize: 0.002
—dataset_shuffle: True
—datasetshuffleseed: 42
—datasettrustremote_code: False
—gradientaccumulationsteps: 1
—weight_decay: 0.0
—maxgradnorm: 1.0
—warmup_ratio: 0.1
—warmup_steps: 0
—gradient_checkpointing: True

</details> <br/>

Framework Versions

—Distily 0.5.0
—Transformers 4.45.0.dev0
—Pytorch 2.5.0.dev20240910+cu121
—Datasets 2.21.0