Hyukkyu/Qwen3-8B-Base-RAQUEL-TOFU-M-ret-LoRA-v1

Qwen3-8B-Base-RAQUEL-TOFU-M-ret-LoRA-v1

A 15-epoch LoRA-trained M_ret baseline for the RAQUEL TOFU reproduction campaign. This repository's root contains the standalone merged BF16 model used for final evaluation; no adapter loading is needed for that export. The original FP32 LoRA adapter is also provided in adapter/.

Mret starts independently from the pinned pretrained base and trains on retain90 biographies and QA only. It does not start from Morig.

This is the LoRA-baseline variant of the paper protocol. It is a baseline checkpoint, not an unlearned model. Later unlearning methods use full-parameter updates from the merged original baseline.

Final evaluation

All table rows were rejudged with pinned Qwen/Qwen3.8-27B at temperature 0 with thinking disabled. Native and paraphrased cohorts contain 400 queries each. RAQUEL accuracy covers every affected query (1,316) and every unaffected query (2,518), with no sampling. The questions were excluded from configuration tuning. Earlier judge results remain in evaluation.json for provenance.

Training

• —Base: Qwen/Qwen3-8B-Base, revision 49e3418fbbbca6ecbdf9608b4d22e5a407081db4.
• —Released RAQUEL2 data: Hyukkyu/RAQUEL2-ICLR, revision aee9a541f1ba58032e2eb62a1e56149475cd1126; native TOFU uses the frozen campaign inputs derived from locuslab/TOFU.
• —Epochs: 15; learning rate: 1e-4; cosine decay; warmup: 3%; weight decay: 0.01.
• —LoRA: rank 64, alpha 128, dropout 0.05; q/k/v/o/gate/up/down projections.
• —Global batch: 32; seed: 0; BF16 base and FP32 adapter training.
• —QA format: Question: {question}\nAnswer:; answer-only QA loss; masked right padding trimmed without removing supervised tokens.

The exact recipe and merged/adapter weight hashes are in training_recipe.json. package_versions.json records the local runtime used to prepare this release. The adapter config uses the public base repository and its pinned revision rather than a machine-local path.

Load the evaluated merged model

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "Hyukkyu/Qwen3-8B-Base-RAQUEL-TOFU-M-ret-LoRA-v1"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
    repo_id, dtype=torch.bfloat16, device_map="auto"
).eval()
prompt = "Question: What is the author's full name?\nAnswer:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.inference_mode():
    output = model.generate(
        **inputs, do_sample=False, max_new_tokens=96,
        pad_token_id=tokenizer.eos_token_id,
    )
print(tokenizer.decode(output[0, inputs.input_ids.shape[1]:], skip_special_tokens=True))

Use the plain QA prefix above for benchmark evaluation. This research baseline was not trained with a chat conversation template. Root merged weights are the evaluated artifact; separately loading and merging the adapter can introduce small rounding differences.

The adapter may be loaded with PeftModel.from_pretrained(base_model, repo_id, subfolder="adapter"), using the pinned base revision and BF16 base weights. Only model/tokenizer files, the portable adapter, aggregate evaluation, recipe, package versions, and upstream notices are included.