MohammadKhosravi/llama3.1-8b-prefixmemory-cefr-balanced-topic-aligned
Low Rank PrefixMemory-Tuning: Decoupled CEFR Baseline
This repository contains the PrefixMemory-Tuning (PMT) baseline developed for controllable CEFR-level text generation.
The model extends Meta Llama-3.1-8B-Instruct with a lightweight external PrefixMemory controller while keeping all base LLM parameters frozen. Only the PrefixMemory controller is optimized during training.
This model serves as the baseline architecture against which all subsequent hierarchical and dual-stream ablation models are compared.
Base Model
Foundation Model
- Model: Meta-Llama-3.1-8B-Instruct
- Parameters: 8 Billion
- Backbone: Decoder-only Transformer
- Transformer Layers: 32
- Hidden Dimension: 4096
- Frozen during training: Yes
Only the PrefixMemory controller is trainable.
Architecture
The architecture follows the PrefixMemory-Tuning paradigm by attaching a lightweight external memory controller to every transformer attention block.
<p align="center"> <img src="images/PMT_Arc.png" width="800"> </p>
<p align="center"> <b>Figure 1.</b> Overall architecture of the CEFR controller integrated with PrefixMemory.. </p>
Controller Pipeline
For a target CEFR level
$$ c \in \{A1, A2, B1, B2, C1, C2\} $$
the controller first retrieves an embedding
$$ e_c = \mathrm{Embedding}(c) $$
which is transformed by a two-layer MLP
$$ m = \mathrm{MLP}(e_c) $$
The modulation vector is reshaped into
$$ m \in \mathbb{R}^{32 \times r} $$
where
- 32 = transformer layers
- r = PrefixMemory rank (64)
For transformer layer $l$, query states are projected into the low-rank PrefixMemory space
$$ Ql = XW^{\mathrm{down}}l $$
where
- $X$ is the hidden representation.
- $W^{\mathrm{down}} \in \mathbb{R}^{4096 \times 64}$.
The controller applies element-wise modulation
$$ \phi(Ql) = \mathrm{ELU}\!\left(Ql \odot m_l\right) $$
The external PrefixMemory bias becomes
$$ Bl = \phi(Ql)W^{\mathrm{up}}_l $$
where
$$ W^{\mathrm{up}} \in \mathbb{R}^{64 \times 4096} $$
Finally,
$$ Hl^{\mathrm{out}} = Hl^{\mathrm{attn}} + \alpha B_l $$
where
- $H_l^{\mathrm{attn}}$ is the original attention output.
- $\alpha$ is a learnable scaling parameter.
Training Objective
Only the PrefixMemory controller parameters are optimized.
The frozen Llama backbone receives the modified hidden representations.
Training minimizes the standard causal language modeling objective
$$ \mathcal{L} = - \sumt \log P(xt \mid x_{<t}, c) $$
using masked prompt tokens.
Optimization
- Optimizer: AdamW
- Learning rate: $$2 \times 10^{-4}$$
- Batch size: 8
- Epochs: 5 ---
Training Dataset
Training uses a balanced subset derived from the EFCAMDAT learner corpus.
Original dataset:
https://ef-lab.mmll.cam.ac.uk/EFCAMDAT.html
Due to the EFCAMDAT data usage policy, the processed training subset cannot be redistributed.
Dataset Columns
Distribution
Balanced across all CEFR levels.
Total:
6,000 samples
Training Performance
Best checkpoint:
Epoch 2
Validation Perplexity:
7.08
[
{
"epoch": 1,
"train_loss": 2.1147,
"val_loss": 2.0288,
"val_ppl": 7.61
},
{
"epoch": 2,
"train_loss": 1.8625,
"val_loss": 1.9573,
"val_ppl": 7.08
},
{
"epoch": 3,
"train_loss": 1.5514,
"val_loss": 2.0191,
"val_ppl": 7.53
},
{
"epoch": 4,
"train_loss": 1.1617,
"val_loss": 2.2208,
"val_ppl": 9.21
},
{
"epoch": 5,
"train_loss": 0.7790,
"val_loss": 2.5407,
"val_ppl": 12.69
}
]In-Domain Evaluation
Benchmark
Balanced benchmark constructed from held-out EFCAMDAT prompts.
- Total prompts: 702
- Source: EFCAMDAT
- Prompt-only evaluation
- Topic distribution balanced across CEFR levels
Dataset columns
Distribution
Results
PMT Evaluation Results
============================================================
📊 FINAL COMPILED MACRO-STATISTICS (PREFIXMEMORY-TUNING BASELINE)
============================================================
Total Processed : 702
Strict Accuracy : 56.27%
Adjacent Accuracy : 73.36%
Mean Abs Error (MAE) : 0.9103
Avg MDD Score : 1.77
Avg Reading Ease : 83.51
Avg Sentence Drift : 2.30 levels
Avg Perplexity : (Computed externally)
============================================================
📈 PERFORMANCE BREAKDOWN BY CEFR TARGET LEVEL
============================================================
strict_accuracy avg_mdd avg_flesch avg_drift
target_cefr
-------------------------------------------------------------
A1 74.36 1.35 89.07 1.01
A2 69.23 1.67 87.02 1.73
B1 60.68 1.85 83.37 2.07
B2 50.43 1.87 82.56 2.94
C1 45.30 1.93 78.48 2.86
C2 37.61 1.95 80.55 3.20
============================================================
📝 DETAILED CLASSIFICATION REPORT
============================================================
precision recall f1-score support
A1 0.56 0.74 0.64 117
A2 0.57 0.69 0.62 117
B1 0.46 0.61 0.53 117
B2 0.48 0.50 0.49 117
C1 0.73 0.45 0.56 117
C2 0.77 0.38 0.51 117
-------------------------------------------------------------
Accuracy 0.56 702
Macro Avg 0.60 0.56 0.56 702
Weighted Avg 0.60 0.56 0.56 702PMT Confusion Matrix
<p align="center"> <img src="images/PMT-Confusion-Matrix-700-Prompt.png" width="800"> </p>
<p align="center"> <b>Figure 2.</b> Confusion Matrix of PrefixMemory Tuning on In-Domain Evaluation Dataset (Inference). </p>
Improvement over Base Llama
- +31.63 percentage points Strict Accuracy
- +10.17 percentage points Adjacent Accuracy
- Lower MAE
- Lower syntactic deviation
- Much stronger readability control
Base LLM Evaluation Results
Baseline LLM Evaluation Results
============================================================
📊 FINAL COMPILED MACRO-STATISTICS (BASE LLM)
============================================================
Total Processed : 690
Strict Accuracy : 24.64%
Adjacent Accuracy : 63.19%
Mean Abs Error (MAE) : 1.3275
Avg MDD Score : 2.00
Avg Reading Ease : 49.47
Avg Sentence Drift : 3.04 levels
Avg Perplexity : 4.14
============================================================
📈 PERFORMANCE BREAKDOWN BY CEFR TARGET LEVEL
============================================================
strict_accuracy avg_mdd avg_flesch avg_drift avg_perplexity
target_cefr
-------------------------------------------------------------------------
A1 16.24 1.73 82.56 3.25 5.16
A2 23.08 1.64 85.44 3.27 5.12
B1 17.39 2.03 57.80 3.11 3.86
B2 41.74 2.15 34.18 2.97 3.32
C1 33.63 2.24 18.95 2.94 3.59
C2 15.93 2.25 15.54 2.69 3.74
============================================================
📝 DETAILED CLASSIFICATION REPORT
============================================================
precision recall f1-score support
A1 0.28 0.16 0.20 117
A2 0.38 0.23 0.29 117
B1 0.21 0.17 0.19 115
B2 0.22 0.42 0.29 115
C1 0.24 0.34 0.28 113
C2 0.23 0.16 0.19 113
-------------------------------------------------------------
Accuracy 0.25 690
Macro Avg 0.26 0.25 0.24 690
Weighted Avg 0.26 0.25 0.24 690Base LLM Confusion Matrix
<p align="center"> <img src="images/basellmconfusion_matrix.png" width="800"> </p>
<p align="center"> <b>Figure 3.</b> Confusion Matrix of Base LLM on In-Domain Evaluation Dataset (Inference). </p>
Out-of-Domain Evaluation
Benchmark
MohammadKhosravi/ielts-cefr-benchmark-prompts
A manually curated benchmark containing IELTS-style prompts designed to evaluate prompt generalization.
Dataset:
https://huggingface.co/datasets/MohammadKhosravi/ielts-cefr-benchmark-prompts
Columns
Distribution
Balanced
Total:
300 prompts
Results
Improvement over Base Llama
- +25.67 percentage points Strict Accuracy
- +17.34 percentage points Adjacent Accuracy
- Lower MAE
- Significantly improved readability control
- Strong transfer to an unseen prompt style
Intended Use
This model is intended for research on
- controllable text generation
- CEFR-level writing generation
- educational NLP
- readability control
- parameter-efficient fine-tuning
- PrefixMemory-Tuning
Citation
If you use this model, please cite:
- PrefixMemory-Tuning (arXiv:2506.13674)
- Meta Llama-3.1-8B-Instruct
- EFCAMDAT Corpus
- This repository