ernlavr/Alpaca-Llama3.1-KD

Dataset Card for Alpaca-Llama3.1-KD

This dataset was introduced in the paper SigmaScale: LLM Compression with SVD-based Low-Rank Decomposition and Learned Scaling Matrices.

The official code repository can be found here: ernlavr/SigmaScale.

Dataset Summary

This dataset is a distilled version of the classic tatsu-lab/alpaca dataset. It utilizes Meta-Llama-3.1-8B-Instruct as an answer generation model to generate high-quality, instruction-following responses for the original 52,000 instructions.

The primary goal of this dataset is to provide a set of responses that align with the Llama 3.1 distribution for model realignment after compression.

Dataset Structure

Each entry in the dataset represents a single prompt from Alpaca, augmented with a Llama 3.1-generated response. For every original prompt, 3 versions (retries) are generated to provide diversity.

• —id Unique identifier for the original prompt.
• —retry_count The iteration index (0-2) for the specific prompt.
• —instruction The task or question provided to the model.
• —input Optional context or data to support the instruction.
• —output_llama The synthetic response generated by Meta-Llama-3.1-8B-Instruct.
• —output_original The original response from the tatsu-lab/alpaca dataset (GPT-3.5 generated).
• —text The full formatted prompt string (including system prompt and chat template) used to prompt the teacher model.

Generation Methodology

The dataset was constructed using the following technical setup:

• —Teacher Model: meta-llama/Meta-Llama-3.1-8B-Instruct

• —Template: The script dynamically extracts the system prompt from the original Alpaca text field and applies the official Llama 3.1 Chat Template.

• —Sampling Parameters:
• —Temperature: 0.7
• —Top-p: 0.9
• —Repetition Penalty: 1.1
• —Max New Tokens: 1024
• —Hardware: Generated using bfloat16 precision for numerical stability.

Prompt Template

The model was prompted using the following structure:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

{extracted_alpaca_system_prompt}<|eot_id|><|start_header_id|>user<|end_header_id|>

### Instruction:
{instruction}

### Input:
{input}

### Response:<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Citing

If you find this work useful, please consider citing

@misc{lavrinovics2026,
      title={SigmaScale: LLM Compression with SVD-based Low-Rank Decomposition and Learned Scaling Matrices}, 
      author={Ernests Lavrinovics and Marco Letizia and Roy Janco and Shai Segal and Johannes Bjerva and Maurizio Pierini},
      year={2026},
      eprint={2606.07098},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2606.07098}, 
}