Wonje/llada2-mini-a2d-lora-kmmlu-safety

LLaDA2.0-mini A2D LoRA

LoRA adapter for inclusionAI/LLaDA2.0-mini, trained with KMMLU Korean multiple-choice data and BeaverTails-style A2D safety-alignment examples.

This repository contains the final PEFT adapter only. Training checkpoints are intentionally omitted from upload.

Training

• —Base model: inclusionAI/LLaDA2.0-mini
• —Method: LoRA / PEFT
• —Target modules: query_key_value, dense
• —LoRA rank: 32
• —LoRA alpha: 64
• —LoRA dropout: 0.05
• —Trainable parameters: 5,898,240
• —Training records: 57,185
• —KMMLU reconstruct records: 30,000
• —BeaverTails A2D EOS records: 15,582
• —BeaverTails reconstruct records: 11,603
• —Effective batch size: 16
• —Optimizer steps: 3,575
• —Max length: 512
• —Max response length: 256
• —Learning rate: 5e-5
• —Weight decay: 0.1
• —Scheduler: cosine
• —Warmup steps: 107
• —Seed: 42
• —Finished: 2026-08-19 09:40:59 UTC

Evaluation

Evaluation was run with the same local benchmark setup used for the pre-LoRA baseline.

Baseline references from the same setup:

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model = "inclusionAI/LLaDA2.0-mini"
adapter_id = "Wonje/llada2-mini-a2d-lora-kmmlu-safety"

tokenizer = AutoTokenizer.from_pretrained(base_model, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    base_model,
    trust_remote_code=True,
    torch_dtype="auto",
    device_map="auto",
)
model = PeftModel.from_pretrained(model, adapter_id)

Use the base model's diffusion decoding/inference code when generating from this adapter.

Limitations

This adapter was trained for a specific Korean QA plus safety-alignment experiment. It should be re-evaluated before use in production or safety-critical settings. KMMLU accuracy decreased in this run, while HarmBench official ASR improved substantially.

Files

• —adapter_model.safetensors: final LoRA weights
• —adapter_config.json: PEFT adapter configuration
• —tokenizer/chat-template files copied for convenience
• —training_summary.json: local training metadata