CoolFace
Modelpublic

sagnikdas-ml/openllama-13b-instruct-human-feedback

sourceHugging Facemitupdated 9mo agoView on Hugging Face
0likes11downloads
Model Card

LoRA OpenLLaMA 13B (HH-RLHF)

Lightweight LoRA adapter for openlm-research/open_llama_13b trained on Anthropic HH-RLHF preference data to encourage helpful, harmless text generation.

Model Details

  • —Developed by: Sagnik Das
  • —Shared by: Sagnik Das
  • —Model type: Causal LM LoRA adapter
  • —Language(s): English
  • —License: MIT
  • —Finetuned from: openlm-research/open_llama_13b
  • —Training data: Anthropic/hh-rlhf (chosen/rejected pairs converted to quality labels)

Model Sources

  • —Repository: https://github.com/sagnik-sudo/LLM-LoRA-Implementation (training code)
  • —Demo: Load with PeftModel.from_pretrained (see below)
  • —Paper: None (work in progress)

Uses

Direct Use

Attach this adapter to the base model for instruction-style text generation, quality-sensitive completions, and safety-aligned responses.

Downstream Use

Can be further tuned or evaluated for safety/quality scoring tasks using the same prompt/quality format.

Out-of-Scope Use

Avoid deployment in safety-critical, high-stakes, or fully autonomous settings without thorough alignment, red-teaming, and monitoring.

Bias, Risks, and Limitations

  • —Trained on HH-RLHF; inherits its coverage and biases.
  • —No safety finetuning beyond dataset intent; harmful or incorrect generations remain possible.
  • —Single-GPU LoRA run; not benchmarked for robustness.

Recommendations

Add task-specific filtering/guardrails and evaluate on your downstream data before production use.

How to Get Started

python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = "openlm-research/open_llama_13b"
adapter = "sagnik-das/lora-openllama-13b-hh-rlhf"

tokenizer = AutoTokenizer.from_pretrained(base, use_fast=False, legacy=False)
tokenizer.pad_token = tokenizer.eos_token
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, adapter)

prompt = "Explain why good documentation matters."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=128, do_sample=True, temperature=0.7, top_p=0.9)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Training Details

  • —Objective: Supervised LM with labels derived from HH-RLHF chosen/rejected pairs ("Quality: good" vs "Quality: bad").
  • —LoRA config: r=8, alpha=16, dropout=0.05; target modules qproj, kproj, vproj, oproj, gateproj, upproj, down_proj; bias=none.
  • —Optimization: AdamW (Trainer defaults), lr=2e-4, weight decay=0.01, warmup=100 steps, maxsteps=7000, gradientaccumulation_steps=16.
  • —Batching: perdevicetrainbatchsize=1, perdeviceevalbatchsize=1.
  • —Precision: bf16 (fallback to fp32 on CPU).
  • —Seq length: 256 tokens, pad to max length.
  • —Checkpoints: saved every 1500 steps; final adapter saved after step 7000.
  • —Hardware: single NVIDIA A40 GPU.

Evaluation

No formal eval recorded; a simple internal win-rate metric over a subset of HH-RLHF was printed during training. Please benchmark on your target tasks before deployment.

Environmental Impact

Not measured; single A40 session, <1 epoch over HH-RLHF with max 7000 steps. Estimate with MLCO2 if required.

Model Card Authors

  • —Sagnik Das