Datasetpublic

donghyunli/Meta-Llama-3-70B-KronQ-HG

Meta-Llama-3-70B — KronQ H_G (output-side gradient covariance) Paper: arXiv:2607.07964 · Code: GitHub Pre-computed H_G for Meta-Llama-3-70B, the output-side curvature factor used by KronQ under the K-FAC factorization H ≈ H_X ⊗ H_G. Per-sublayer empirical-Fisher gradient covariance (E[g gᵀ] over the layer output), distinct from the input-side Hessian H_X. Publishing this lets you reproduce KronQ quantization without the offline Fisher precompute step. Contents (80… See the full description on the dataset page: https://huggingface.co/datasets/donghyunli/Meta-Llama-3-70B-KronQ-HG.

sourceHugging Facellama3updated 3mo agoView on Hugging Face

0likes2.1kdownloads

Dataset Card

Meta-Llama-3-70B — KronQ H_G (output-side gradient covariance)

Paper: arXiv:2607.07964 · Code: GitHub

Pre-computed H_G for Meta-Llama-3-70B, the output-side curvature factor used by KronQ under the K-FAC factorization H ≈ H_X ⊗ H_G. Per-sublayer empirical-Fisher gradient covariance (E[g gᵀ] over the layer output), distinct from the input-side Hessian H_X.

Publishing this lets you reproduce KronQ quantization without the offline Fisher precompute step.

Contents (80 layers × 7 sublayers, ~551 GB, fp32)

layer_<i>/self_attn_{q,k,v,o}_proj_G.pt
layer_<i>/mlp_{gate,up,down}_proj_G.pt
metadata.pt

Each *_G.pt is the out_features × out_features gradient covariance for that sublayer.

Note: this is the rotated HG (QuaRot frame). For the per-channel weight-only recipe, HG cancels in the OBS update, so rotated and raw give identical quantized weights — use directly with --grad_dir.

Usage

bash

python main.py --model meta-llama/Meta-Llama-3-70B \\
    --w_bits 4 --w_groupsize -1 --w_clip --w_asym --a_bits 16 --act_order \\
    --bi_calibration --use_gptaq --incoh_rotate --incoh_kernel had --incoh_mode full \\
    --alpha 0.25 --grad_dir <downloaded_HG_dir>

License

Derived from Meta-Llama-3-70B — llama3 license.