Modelpublic

tuongvy2603/continue_sft_bitd_lora_full_data_k01

sourceHugging Faceapache-2.0updated 4mo agoView on Hugging Face

0likes5downloads

Model Card

continuesftbitdlorafulldatak01

LoRA adapter trained on top of `tuongvy2603/BITD_baseline` — a second-stage continued SFT on the full_data / k=1 data pool. Loading base + this adapter gives a model that has been SFT'd twice.

This adapter is part of a sweep over the per-prompt sample budget k used to build the continued-SFT training set from the full data pool.

Usage

Install

bash

pip install transformers peft torch

Load and run

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# 1. Load base model
base = AutoModelForCausalLM.from_pretrained(
    "tuongvy2603/BITD_baseline",
    dtype=torch.bfloat16,
    device_map="auto",
)

# 2. Attach LoRA adapter
model = PeftModel.from_pretrained(base, "tuongvy2603/continue_sft_bitd_lora_full_data_k01")

# 3. Tokenizer (chat template lives here)
tok = AutoTokenizer.from_pretrained("tuongvy2603/continue_sft_bitd_lora_full_data_k01")

# 4. Generate
messages = [{"role": "user", "content": "Pick open-minded or close-minded."}]
inputs = tok.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)
out = model.generate(inputs, max_new_tokens=64, do_sample=False)
print(tok.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))

No merging required — PeftModel injects the LoRA weights at runtime. Forward results are mathematically identical to a merged model.

Optional: merge for standalone use

If you want a single self-contained checkpoint (e.g. for llama.cpp / GGUF conversion, or serving stacks that don't load adapters):

python

merged = model.merge_and_unload()
merged.save_pretrained("continue_sft_bitd_lora_full_data_k01_merged")
tok.save_pretrained("continue_sft_bitd_lora_full_data_k01_merged")

Training details


Base model	`tuongvy2603/BITD_baseline`
Method	LoRA (PEFT)
Data pool	`full_data`, `k = 1` samples / prompt
LoRA rank / alpha / dropout	16 / 32 / 0.05
Target modules	`q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj`
Epochs	5.0
Batch size	8 × 2 grad accum = 16 effective
Learning rate	0.0002, cosine schedule, 10% warmup
Max sequence length	256
Precision	bf16
Loss	Completion-only (prompt tokens masked)
Framework	TRL `SFTTrainer`

Full resolved config: see run_config.json in this repo.

Framework versions

—PEFT 0.19.1
—TRL 1.4.0
—Transformers 5.8.1
—PyTorch 2.12.0
—Datasets 4.8.5
—Tokenizers 0.22.2

Citation

bibtex

@software{vonwerra2020trl,
  title   = {{TRL: Transformers Reinforcement Learning}},
  author  = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallouédec, Quentin},
  license = {Apache-2.0},
  url     = {https://github.com/huggingface/trl},
  year    = {2020}
}