AnonymousSubmission9090/llama-alpaca

Dataset Card for Alpaca-Llama3.1-KD

Dataset Summary

This dataset is a distilled version of the classic tatsu-lab/alpaca dataset. It utilizes Meta-Llama-3.1-8B-Instruct as an answer generation model to generate high-quality, instruction-following responses for the original 52,000 instructions.

The primary goal of this dataset is to provide a set of responses that align with the Llama 3.1 distribution for model realignment after compression

Dataset Structure

Each entry in the dataset represents a single prompt from Alpaca, augmented with a Llama 3.1-generated response. For every original prompt, 3 versions (retries) are generated to provide diversity.

• —id Unique identifier for the original prompt.
• —retry_count The iteration index (0-2) for the specific prompt.
• —instruction The task or question provided to the model.
• —input Optional context or data to support the instruction.
• —output_llama The synthetic response generated by Meta-Llama-3.1-8B-Instruct.
• —output_original The original response from the tatsu-lab/alpaca dataset (GPT-3.5 generated).
• —text The full formatted prompt string (including system prompt and chat template) used to prompt the teacher model.

Generation Methodology

The dataset was constructed using the following technical setup:

• —meta-llama/Meta-Llama-3.1-8B-Instruct

Template: The script dynamically extracts the system prompt from the original Alpaca text field and applies the official Llama 3.1 Chat Template.

Sampling Parameters:

• —Temperature: 0.7
• —Top-p: 0.9
• —Repetition Penalty: 1.1
• —Max New Tokens: 1024
• —Hardware: Generated using bfloat16 precision for numerical stability.

Prompt Template

The model was prompted using the following structure:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

{extracted_alpaca_system_prompt}<|eot_id|><|start_header_id|>user<|end_header_id|>

### Instruction:
{instruction}

### Input:
{input}

### Response:<|eot_id|><|start_header_id|>assistant<|end_header_id|>