Modelpublic

elastix-ai/Qwen2.5-0.5B-maskllm-beam-2to4-calib512-weakscale-blade

sourceHugging Faceupdated 1mo agoView on Hugging Face

0likes43downloads

Model Card

Compressed: Qwen2.5-0.5B-maskllm-beam-2to4-calib512-weakscale-blade

This is a compressed version of `Qwen/Qwen2.5-0.5B`.

Compression Summary

Field	Value
Original model	`Qwen/Qwen2.5-0.5B`
Pruning method	blade
Calibration samples	512
Calibration sequence length	2048
Calibration dataset	`{"path": "DKYoon/SlimPajama-6B", "split": "validation", "streaming": true}`
Symmetric calibration	False
BEAM fine-tuning	True
Convert experts to linear	False
GFP checkpoint format	False
MLflow run	e76135610e074fb686ccae0fc94edbd2

Layerwise Compression Specs

Pattern	Sparsity	Quantize	Bits	Type	Groupsize	Quant Dim
`.*embed_tokens$`	0	no	16	gfp	32	-1
`.*lm_head$`	0	no	16	gfp	32	-1
`(?:^	\.)linear_attn\.conv1d$`	0	no	16	gfp	32	-1
`(?:^	\.)(?:self_attn	linear_attn)\.`	2:4	no	16	gfp	32	-1
`(?:^	\.)mlp\.router$`	0	no	16	gfp	32	-1
`.*`	2:4	no	16	gfp	32	-1

json

[
  {
    "pattern": ".*embed_tokens$",
    "quant_config": {
      "bits": 16,
      "type": "gfp",
      "groupsize": 32,
      "exp_bits": 5,
      "exp_bias": 18,
      "expfact_m_bits": 3,
      "expfact_m_range_oct": 2.0,
      "scale_search": {
        "grid_search_steps": 5,
        "grid_search_min_ratio": 0.3,
        "ls_polish_steps": 2,
        "clip_ratio": 1.0
      },
      "qtip": {
        "L": 16,
        "T": null,
        "hash_bytes": 4,
        "kappa": null,
        "hash_seed": 0,
        "boundary": "prologue",
        "candidates": 4
      },
      "rotate_seed": 0
    },
    "sparsity": 0,
    "quantize": false,
    "quantize_before_pruning": false,
    "quant_dim": -1
  },
  {
    "pattern": ".*lm_head$",
    "quant_config": {
      "bits": 16,
      "type": "gfp",
      "groupsize": 32,
      "exp_bits": 5,
      "exp_bias": 18,
      "expfact_m_bits": 3,
      "expfact_m_range_oct": 2.0,
      "scale_search": {
        "grid_search_steps": 5,
        "grid_search_min_ratio": 0.3,
        "ls_polish_steps": 2,
        "clip_ratio": 1.0
      },
      "qtip": {
        "L": 16,
        "T": null,
        "hash_bytes": 4,
        "kappa": null,
        "hash_seed": 0,
        "boundary": "prologue",
        "candidates": 4
      },
      "rotate_seed": 0
    },
    "sparsity": 0,
    "quantize": false,
    "quantize_before_pruning": false,
    "quant_dim": -1
  },
  {
    "pattern": "(?:^|\\.)linear_attn\\.conv1d$",
    "quant_config": {
      "bits": 16,
      "type": "gfp",
      "groupsize": 32,
      "exp_bits": 5,
      "exp_bias": 18,
      "expfact_m_bits": 3,
      "expfact_m_range_oct": 2.0,
      "scale_search": {
        "grid_search_steps": 5,
        "grid_search_min_ratio": 0.3,
        "ls_polish_steps": 2,
        "clip_ratio": 1.0
      },
      "qtip": {
        "L": 16,
        "T": null,
        "hash_bytes": 4,
        "kappa": null,
        "hash_seed": 0,
        "boundary": "prologue",
        "candidates": 4
      },
      "rotate_seed": 0
    },
    "sparsity": 0,
    "quantize": false,
    "quantize_before_pruning": false,
    "quant_dim": -1
  },
  {
    "pattern": "(?:^|\\.)(?:self_attn|linear_attn)\\.",
    "quant_config": {
      "bits": 16,
      "type": "gfp",
      "groupsize": 32,
      "exp_bits": 5,
      "exp_bias": 18,
      "expfact_m_bits": 3,
      "expfact_m_range_oct": 2.0,
      "scale_search": {
        "grid_search_steps": 5,
        "grid_search_min_ratio": 0.3,
        "ls_polish_steps": 2,
        "clip_ratio": 1.0
      },
      "qtip": {
        "L": 16,
        "T": null,
        "hash_bytes": 4,
        "kappa": null,
        "hash_seed": 0,
        "boundary": "prologue",
        "candidates": 4
      },
      "rotate_seed": 0
    },
    "sparsity": "2:4",
    "quantize": false,
    "quantize_before_pruning": false,
    "quant_dim": -1
  },
  {
    "pattern": "(?:^|\\.)mlp\\.router$",
    "quant_config": {
      "bits": 16,
      "type": "gfp",
      "groupsize": 32,
      "exp_bits": 5,
      "exp_bias": 18,
      "expfact_m_bits": 3,
      "expfact_m_range_oct": 2.0,
      "scale_search": {
        "grid_search_steps": 5,
        "grid_search_min_ratio": 0.3,
        "ls_polish_steps": 2,
        "clip_ratio": 1.0
      },
      "qtip": {
        "L": 16,
        "T": null,
        "hash_bytes": 4,
        "kappa": null,
        "hash_seed": 0,
        "boundary": "prologue",
        "candidates": 4
      },
      "rotate_seed": 0
    },
    "sparsity": 0,
    "quantize": false,
    "quantize_before_pruning": false,
    "quant_dim": -1
  },
  {
    "pattern": ".*",
    "quant_config": {
      "bits": 16,
      "type": "gfp",
      "groupsize": 32,
      "exp_bits": 5,
      "exp_bias": 18,
      "expfact_m_bits": 3,
      "expfact_m_range_oct": 2.0,
      "scale_search": {
        "grid_search_steps": 5,
        "grid_search_min_ratio": 0.3,
        "ls_polish_steps": 2,
        "clip_ratio": 1.0
      },
      "qtip": {
        "L": 16,
        "T": null,
        "hash_bytes": 4,
        "kappa": null,
        "hash_seed": 0,
        "boundary": "prologue",
        "candidates": 4
      },
      "rotate_seed": 0
    },
    "sparsity": "2:4",
    "quantize": false,
    "quantize_before_pruning": false,
    "quant_dim": -1
  }
]

</details>

BEAM Fine-tuning

Field	Value
Ray Tune enabled	False
Num samples	20
Max steps	100
Min steps	10
Reduction factor	3
Search algorithm	optuna
Search space	`{'lr': {'type': 'loguniform', 'min': 1e-05, 'max': 0.01}, 'batch_size': {'type': 'choice', 'values': [8, 16, 32]}, 'weight_decay': {'type': 'uniform', 'min': 0.0, 'max': 0.1}, 'scheduler_warmup_ratio': {'type': 'uniform', 'min': 0.0, 'max': 0.2}, 'steps': 100, 'optimizer': 'adamw', 'scheduler': 'cosine', 'scheduler_min_lr_ratio': 0.1}`

Evaluation Results

KL Divergence

Dataset	Avg KL	Total KL	Tokens
wikitext2	0.379138	2082.9830	5,494
c4	0.345287	12724.5247	36,852