MohammadKhosravi/llama3.1-8b-cefr-steering-1layer-head-ordinal-universal

Llama-3.1-8B CEFR Single-Layer Latent Classifier

This repository contains the CEFR latent attribute classifier used by the PPLM baseline in the master's thesis:

Beyond Prompting: Resource-Efficient Explicit Control for CEFR-Aligned Language Generation

The classifier operates on hidden-state representations extracted from meta-llama/Llama-3.1-8B-Instruct and predicts one of the six Common European Framework of Reference for Languages (CEFR) proficiency levels:

A1, A2, B1, B2, C1, and C2.

Its role is not standalone text generation. Instead, it provides the differentiable CEFR objective used by the Plug and Play Language Model (PPLM) baseline during inference-time steering.

Role in the PPLM Pipeline

The PPLM pipeline uses this classifier as an attribute model over internal Llama-3.1-8B representations.

Llama-3.1-8B hidden representation
                ↓
        Dropout (p = 0.35)
                ↓
         Linear(4096 → 6)
                ↓
          CEFR class logits
        A1 A2 B1 B2 C1 C2

During PPLM generation, the CEFR classification objective provides a differentiable signal that can be propagated with respect to the model's latent representation to steer generation toward a requested CEFR level.

Architecture

The classifier is intentionally lightweight:

• —Input dimension: 4096
• —Input representation: Llama-3.1-8B hidden-state vector
• —Dropout: 0.35
• —Classifier: Single linear projection
• —Output dimension: 6
• —Output classes: A1, A2, B1, B2, C1, C2

The model can be summarized as:

Dropout(0.35) → Linear(4096, 6)

No intermediate hidden layer, nonlinear activation, or LayerNorm module is used.

The shallow architecture was selected to keep the attribute model lightweight and fully differentiable during PPLM decoding.

Training Dataset

The classifier is trained on the latent CEFR dataset:

MohammadKhosravi/cefr-llama3.1-8b-hidden-states-efcamdat-universal

The dataset contains 66,494 latent representations paired with CEFR labels.

Each sample consists of:

• —A 4096-dimensional hidden-state representation
• —A CEFR target label from A1 to C2

The latent dataset includes representations constructed from the EFCAMDAT Steering Training Dataset together with the pre-existing CEFR latent dataset used in the PPLM experiment.

The construction pipeline is available in the thesis GitHub repository:

notebooks/01_data_preparation/pplm_latent_dataset_construction.ipynb

Train / Validation Split

The full latent dataset is divided using a stratified split:

• —Training: 85%
• —Validation: 15%
• —Random seed: 42

The resulting training partition contains 56,519 latent vectors, with the remaining samples used for validation.

Stratification preserves the CEFR class distribution across the two partitions.

Class-Imbalance Handling

The CEFR latent dataset is strongly imbalanced, particularly for the advanced C1 and C2 levels.

Inverse-frequency class weights are computed from the training partition using sklearn.utils.class_weight.compute_class_weight.

The resulting weights used during training were:

The substantially larger C1 and C2 weights compensate for the much lower representation of advanced-level samples in the latent training data.

Training Objective

Training uses a custom Ordinal Weighted Loss combining weighted cross-entropy with an ordinal distance penalty.

The total objective is:

$$ \mathcal{L} = \mathcal{L}{\mathrm{WCE}} + \lambda{\mathrm{ord}}\mathcal{L}_{\mathrm{ordinal}} $$

where:

• —$\mathcal{L}_{\mathrm{WCE}}$ is weighted cross-entropy using the inverse-frequency CEFR class weights.
• —$\mathcal{L}_{\mathrm{ordinal}}$ is a Mean Squared Error penalty between the expected predicted class index and the true CEFR class index.

This encourages the classifier to account for the ordered structure of the CEFR scale in addition to exact class prediction.

Training Configuration

The learning rate is reduced when validation Strict Accuracy plateaus.

Training Log

The training run produced the following recorded checkpoints:

The training run used 56,519 training vectors.

The trajectory shows rapid improvement during the first 10–15 epochs, followed by a plateau around 84–85% Strict Accuracy, while Adjacent Accuracy remains around 97%.

Best Validation Checkpoint

The model checkpoint is selected according to the highest observed validation Strict Accuracy.

The best result obtained during training was:

The corresponding Adjacent Accuracy is the value recorded at the checkpoint selected by the highest Strict Accuracy; it is not independently selected as the maximum Adjacent Accuracy across epochs.

The training procedure saves a new checkpoint whenever validation Strict Accuracy exceeds the previous best value.

Metric Definitions

Strict Accuracy

Strict Accuracy requires the predicted CEFR level to exactly match the target CEFR class.

Adjacent Accuracy

Adjacent Accuracy counts a prediction as correct when it is either the exact CEFR level or one immediately neighboring level on the ordinal CEFR scale.

For example:

Target B1 → B1       = Strict + Adjacent correct
Target B1 → A2 / B2  = Adjacent correct
Target B1 → A1 / C1 / C2 = Incorrect

The adjacent metric is useful because CEFR is ordinal and classification errors between neighboring levels are less severe than errors spanning several proficiency levels.

Usage in the Thesis

This classifier is used exclusively as the attribute classifier for the PPLM baseline.

The PPLM experiments use gradients derived from the classifier's CEFR objective during inference-time latent perturbation while the parameters of Llama-3.1-8B-Instruct remain frozen.

The classifier itself is trained separately from the generation experiments.

Evaluation of the resulting PPLM-controlled generations is performed in a separate notebook in the thesis repository.

Reproducibility

The classifier-training notebook is available at:

notebooks/03_pplm/pplm_latent_classifier_train.ipynb

The latent-dataset construction notebook is available at:

notebooks/01_data_preparation/pplm_latent_dataset_construction.ipynb

The training notebook retains the original experimental outputs, including:

• —Dataset loading
• —Train-validation splitting
• —Dynamically computed CEFR class weights
• —Epoch-level training losses
• —Learning-rate changes
• —Validation Strict Accuracy
• —Validation Adjacent Accuracy
• —Best-checkpoint selection

Intended Use

This model is intended for research on:

• —PPLM-style inference-time steering
• —CEFR-controlled text generation
• —Latent-space proficiency classification
• —Ordinal CEFR classification
• —Controllable generation with frozen language models

Important: This model is not intended to serve as a standalone general-purpose CEFR assessment model for arbitrary learner texts.

It was trained specifically on Llama-3.1-8B latent representations for use within the PPLM experimental pipeline.

Citation

If you use this model or the accompanying experimental resources, please cite the associated master's thesis and repository.

@mastersthesis{khosravi_cefr_control,
  title  = {Beyond Prompting: Resource-Efficient Explicit Control for CEFR-Aligned Language Generation},
  author = {Mohammad Khosravi},
  school = {University of Padova},
  year   = {2026}
}