Hyukkyu/Llama-3.1-8B-RAQUEL-MUSE-M-orig-LoRA-v1

Llama-3.1-8B-RAQUEL-MUSE-M-orig-LoRA-v1

Built with Llama. See LICENSE, NOTICE, and USE_POLICY.md.

A LoRA-trained M_orig baseline for the RAQUEL MUSE experiments. Morig is trained from the pinned pretrained `meta-llama/Llama-3.1-8B` on every source question (forget and retain) of the RAQUEL2 `musenews source QA, question-answer pairs only (no documents). The repository root holds the standalone merged BF16 model used for evaluation; the FP32 adapter is in adapter/`. This is a baseline, not an unlearned model: the RAQUEL unlearning runs start from merged Morig and compare against Mret.

Release note. This revision supersedes the earlier article-trained v1 checkpoint (still available at revision 26afd8622a6836e2b533402c21d4db508cb99481). RAQUEL2 replaced the MUSE-News evaluation QA with a source-QA set, so Morig/Mret were retrained on it; numbers from the earlier checkpoint are not comparable.

Evaluation

Semantic accuracy judged by Qwen/Qwen3.8-27B (vLLM 0.28.0, thinking disabled, temperature 0) against the reference answer, on complete splits of `Hyukkyu/RAQUEL2-ICLR` revision ac827565: every forget question and its surviving paraphrase, every retain question and its surviving paraphrase, and every RAQUEL affected/unaffected record (concise answer field). Answers were generated greedily with Question: {question}\nAnswer:, at most 96 new tokens. Per-split counts and evidence hashes are in evaluation.json.

Training

• —Base: meta-llama/Llama-3.1-8B, revision d04e592bb4f6aa9cfee91e2e20afa771667e1d4b.
• —Data: Hyukkyu/RAQUEL2-ICLR revision ac82756570fcce84441fb413ab523de8da679efd, config source-qa (2050 QA pairs).
• —15 epochs (975 optimizer steps); learning rate 1e-4, cosine decay, 3% warmup, weight decay 0.01.
• —LoRA rank 64, alpha 128, dropout 0.05 on q/k/v/o/gate/up/down projections; BF16 base, FP32 adapters.
• —Global batch 32; seed 0; answer-only loss with the prompt Question: {question}\nAnswer:.

Exact settings and weight hashes are in training_recipe.json.

Load

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "Hyukkyu/Llama-3.1-8B-RAQUEL-MUSE-M-orig-LoRA-v1"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id, dtype=torch.bfloat16, device_map="auto").eval()
prompt = "Question: {question}\nAnswer:"

The root merged weights are the evaluated artifact. The FP32 LoRA adapter is in adapter/ (PeftModel.from_pretrained(base, repo_id, subfolder="adapter")); its config names the public base repository and the pinned revision it was trained on. Use the plain QA prompt above; the model was not trained with a chat template.