MohammadKhosravi/llama3.1-8b-standard-prefix-tuning-6k

Llama-3.1-8B Standard Prefix-Tuning CEFR Baseline

This repository contains the Standard Prefix-Tuning baseline used in the master's thesis:

Beyond Prompting: Resource-Efficient Explicit Control for CEFR-Aligned Language Generation

The model applies the Hugging Face PEFT implementation of Prefix-Tuning to meta-llama/Llama-3.1-8B-Instruct.

The Llama-3.1-8B-Instruct backbone remains frozen while a continuous learnable prefix is optimized for CEFR-conditioned text generation across the six proficiency levels:

A1, A2, B1, B2, C1, and C2.

This model serves as a generic PEFT baseline for comparison with the CEFR-specific control architectures investigated in the thesis.

Model Overview

Standard Prefix-Tuning learns continuous virtual prefix representations while keeping the underlying language-model parameters frozen.

The prefix representations are injected into the transformer's attention key/value states through the PEFT Prefix-Tuning implementation.

The configuration used in this experiment is:

        Frozen Llama-3.1-8B-Instruct
                     +
            30 Virtual Prefix Tokens
                     +
              Prefix Projection

Prefix-Tuning Configuration

Only the Prefix-Tuning parameters are optimized during training. The Llama-3.1-8B-Instruct backbone remains frozen.

Training Dataset

Training uses the Balanced CEFR Steering Subset derived from the EFCAMDAT Steering Training Dataset.

The dataset contains exactly 5,568 examples, balanced equally across the six CEFR levels:

A stratified 90/10 train-validation split is used with random_state=42:

Training Prompt and Target Construction

Each training example contains:

• —A target CEFR level
• —A writing topic
• —A reference response from the steering dataset

The textual instruction explicitly specifies the requested CEFR level.

The training sequence consists of the formatted instruction followed by the reference response.

Prompt and padding tokens are excluded from the causal language-modeling loss using the standard -100 ignore index, so optimization is performed only on the target response tokens.

Training Objective

Training uses the standard autoregressive causal language-modeling cross-entropy objective.

The backbone parameters remain frozen and gradients update only the learnable Prefix-Tuning parameters.

Conceptually, the target-only objective is:

$$ \mathcal{L}{\text{CLM}} = -\sum{t \in \mathcal{T}{\text{target}}} \log p{\theta}(yt \mid y{<t},x) $$

where $\mathcal{T}_{\text{target}}$ contains only the response-token positions included in the loss.

Training Configuration

Training Progression

The recorded training trajectory is:

Validation performance improves across all three epochs.

Validation loss decreases from:

2.5390 → 2.4701 → 2.4610

while validation perplexity decreases from:

12.67 → 11.82 → 11.72

The lowest validation loss and perplexity are obtained at Epoch 3. The Epoch 3 adapter is therefore retained as the final checkpoint.

Generation Evaluation

The trained Prefix-Tuning adapter is evaluated on the final In-Domain Evaluation Prompt Matrix used throughout the thesis.

The benchmark contains:

117 topics × 6 CEFR levels = 702 generation conditions

For evaluation, the trained Prefix-Tuning adapter is attached to the frozen Llama-3.1-8B-Instruct backbone using PEFT.

Generation Configuration

Generated texts are evaluated using the primary CEFR evaluator:

MohammadKhosravi/roberta-large-cefr-classifier-JointLoss

Evaluation Results

The Standard Prefix-Tuning baseline obtains the following results over the complete 702-condition in-domain evaluation matrix:

Metric Interpretation

Strict Accuracy

A generation is counted as correct only when the predicted CEFR level exactly matches the requested target level.

Adjacent Accuracy

Adjacent Accuracy additionally treats predictions one CEFR level away from the requested target as correct.

For example:

Target B1:

B1           → Strict + Adjacent correct
A2 / B2      → Adjacent correct
A1 / C1 / C2 → Incorrect

Mean Absolute Error

CEFR levels are mapped to ordered class indices:

A1 = 0
A2 = 1
B1 = 2
B2 = 3
C1 = 4
C2 = 5

MAE measures the average ordinal distance between the requested and predicted CEFR levels:

$$ \operatorname{MAE} = \frac{1}{N} \sum{i=1}^{N} \left| \hat{y}i-y_i \right| $$

Lower values indicate smaller CEFR-control errors.

Linguistic Diagnostics

The evaluation additionally reports:

• —Mean Dependency Distance (MDD) as a syntactic-complexity diagnostic
• —Flesch Reading Ease as a readability diagnostic
• —Sentence-level CEFR drift as a measure of proficiency consistency within an individual generated response

These diagnostics complement the primary CEFR-control metrics.

Role in the Thesis

Standard Prefix-Tuning is included as a generic parameter-efficient baseline against which the explicitly CEFR-conditioned methods are compared.

Unlike the proposed CEFR-specific controllers, this baseline does not contain a dedicated architectural CEFR-conditioning mechanism.

The target proficiency level is supplied through the textual instruction, while the same shared learnable prefix mechanism is used across CEFR levels.

This provides a comparison between:

             Generic Prefix-Tuning
                      vs.
      Explicit CEFR-Conditioned Prefix Control

The experiment is therefore useful for separating improvements due to explicit CEFR conditioning from improvements that can be obtained through standard parameter-efficient adaptation alone.

Relationship to PrefixMemory-Tuning

This repository implements the Standard Prefix-Tuning baseline using the PEFT Prefix-Tuning mechanism.

It should be distinguished from the PrefixMemory-Tuning and CEFR-conditioned controller variants evaluated elsewhere in the thesis.

In this baseline:

• —One shared prefix mechanism is trained
• —CEFR information is provided through the textual instruction
• —No dedicated CEFR embedding is introduced
• —No CEFR-specific gating mechanism is introduced

Conceptually:

                  Standard Prefix-Tuning
                           │
             ┌─────────────┴─────────────┐
             │                           │
      CEFR Information             Prefix Parameters
             │                           │
      Textual Instruction          Shared Prefix
             │                           │
             └─────────────┬─────────────┘
                           │
                           ▼
                  Frozen Llama-3.1-8B

This differs from the explicitly conditioned architectures, where CEFR information directly controls dedicated trainable architectural components.

Reproducibility

The complete training and evaluation notebook is available in the thesis GitHub repository at:

notebooks/04_generic_peft_baselines/standard_prefix_tuning_train_and_eval.ipynb

The balanced training subset is constructed in:

notebooks/01_data_preparation/balanced_cefr_steering_subset.ipynb

The in-domain evaluation benchmark is constructed in:

notebooks/01_data_preparation/efcamdat_preprocessing_and_partitioning.ipynb

The experimental notebook retains the original outputs, including:

• —Train/validation split statistics
• —Trainable-parameter counts
• —Epoch-level training losses
• —Validation losses and perplexities
• —Generation outputs
• —CEFR evaluation metrics
• —Classification reports
• —Confusion matrices

Intended Use

This adapter is intended for research on:

• —Prefix-Tuning
• —Parameter-efficient fine-tuning
• —CEFR-controlled text generation
• —Proficiency-conditioned generation
• —Controllable language generation
• —Comparison of generic PEFT methods with explicit control mechanisms

Important: This repository contains an experimental research artifact and is not intended to serve as a standalone CEFR assessment model.

Citation

If you use this model or the accompanying experimental resources, please cite the associated master's thesis and repository.

@mastersthesis{khosravi2026beyond,
  author = {Mohammad Khosravi},
  title = {Beyond Prompting: Resource-Efficient Explicit Control for CEFR-Aligned Language Generation},
  school = {University of Padova},
  year = {2026}
}