elastix-ai/HyperPrune-Llama-3.1-8B-4to8

HyperPrune-Llama-3.1-8B-4to8

meta-llama/Llama-3.1-8B pruned to 4:8 semi-structured sparsity with HyperPrune (Sun & Sakuma, Learning Semi-Structured Sparsity for LLMs via Shared and Context-Aware Hypernetwork, ICLR 2026, OpenReview).

This is a reproduction run produced at Elastix as part of the BLADE sparsity-method comparison. It is plain sparse bf16/fp16 safetensors and loads with stock transformers:

from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("elastix-ai/HyperPrune-Llama-3.1-8B-4to8")
t = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B")

What differs from the paper's own recipe

Everything else — hypernet architecture, both training stages, all learning rates, step counts, temperature, prior, row selection — is HyperPrune's own shipped setting.

Configuration

{
  "model": {
    "name_or_path": "meta-llama/Llama-3.1-8B",
    "tokenizer": "meta-llama/Llama-3.1-8B"
  },
  "data": {
    "dataset_name": "slimpajama",
    "num_samples": 128,
    "seq_len": 2048,
    "seed": 42
  },
  "sparsity": {
    "n": 4,
    "m": 8
  },
  "hypernet": {
    "type": "mlp",
    "hidden_dim": 256,
    "emb_dim": 64,
    "use_layer_emb": false,
    "use_comp_emb": false,
    "use_hessian_diag": true
  },
  "training": {
    "sup_steps": 12000,
    "sup_lr": 0.001,
    "ft_lr": 0.0003,
    "ft_nsamples": 4,
    "rows_per_step": 400,
    "cascade_inner_steps": 300,
    "ft_mode": "cascade",
    "tau": 0.5,
    "prior_source": "sparsegpt",
    "wanda_residual_alpha": 2.0,
    "compensated_propagation": true,
    "use_weight_compensation": true,
    "train_on_compensated": true,
    "fixed_rows_count": 200,
    "fixed_rows_pos": "first",
    "dense_layers_list": []
  },
  "output": {
    "save_dir": "/home/ubuntu/hyperprune_work/outputs/hp-llama31_8b-4to8",
    "wanda_dir": "/home/ubuntu/hyperprune_work/outputs/hp-llama31_8b-4to8_ref",
    "preserve_wanda_dir": false
  }
}

Measured

Two things to know before comparing this number to the paper

1. Every decoder layer is pruned here. HyperPrune's own shipped configs set dense_layers_list: [0, 1], leaving 2 layers fully dense and yielding ~46.9 % sparsity rather than 50 %. This checkpoint prunes every layer, matching BLADE's four_eight_all spec, so it is a true 4:8 model in the modules BLADE prunes.

2. Only a few percent of this mask was chosen by the hypernet. The shipped recipe sets fixed_rows_count: 200, so the hypernet decides the mask for the first 200 output rows of each projection and every remaining row keeps the SparseGPT prior's mask verbatim. This is HyperPrune's own default, kept here deliberately because the brief was to change nothing but the calibration corpus.

The two perplexity rows are different quantities and are not comparable to each other. The first is token-level PPL over concatenated WikiText-2 at seqlen 2048 (the Wanda/SparseGPT convention). The second is lm-evaluation-harness word_perplexity at max_length=2048, which is BLADE's protocol — pinned empirically by reproducing BLADE's dense LLaMA-2-7B value of 9.19 (measured 9.1915).

4:8 is an extension, not a config flag

HyperPrune ships 2:4 only: the group width 4, the kept-count 2 and the 6-way pattern space are hard-coded across ~40 sites. This checkpoint comes from an N:M generalization (ElastiX patch #13) validated by three gates:

1. 1.2:4 stays bit-identical through the generalized code — every per-layer reconstruction value matches the unmodified implementation exactly.
2. 2.4:8 runs end to end with C(8,4)=70 patterns per group.
3. 3.Strict pattern compliance: zero groups anywhere in the checkpoint have more than 4 surviving weights out of 8. (Groups with fewer than 4 exist — kept positions whose weight was already exactly 0.0 in the dense model — which is still valid N:M, since the constraint bounds nonzeros from above.)

4:8 is a markedly easier pattern than 2:4 and the numbers reflect it: this model goes from 22.27 (2:4) to 16.39 word-perplexity.

Provenance

Produced from HyperPrune commit 6d093d7 with a small set of documented patches (bias-dtype autocast, calibration loader, disk-peak reduction, and — for 4:8 checkpoints — the N:M generalization, which the reference implementation does not ship). See the reproduction report for the full diff.

<!-- COMPRESSIONEVALRESULTS:BEGIN -->

Evaluation Results

KL Divergence

Downstream Accuracy

Perplexity (2048-token windows, max_length=2048)

BLADE-Eval: lm-eval 0.4.10, torch 2.13.0+cu130, [MLflow run](https://experiments.external.elastix.ai:9099/#/experiments/90/runs/f7e0b5b323e24c6bb7d75b7ea9e1c05e) <!-- COMPRESSIONEVALRESULTS:END -->