nishantmulchandani/PAMAv1-Nemotron-8B-1M-LoRA

PAMAv1 - LoRA Adapters for nvidia/Llama-3.1-Nemotron-8B-UltraLong-1M-Instruct

Model on HF | Live Space

Summary

• —Goal: Share fine-tuned LoRA adapters for long-context instruction following.
• —Base: nvidia/Llama-3.1-Nemotron-8B-UltraLong-1M-Instruct
• —Weights: Adapters only (no merged base weights).
• —Intended use: R&D, prototyping, and experimentation.

How To Use (Adapters)

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

base = "nvidia/Llama-3.1-Nemotron-8B-UltraLong-1M-Instruct"
adapters = "nishantmulchandani/PAMAv1-Nemotron-8B-1M-LoRA"

tok = AutoTokenizer.from_pretrained(base, trust_remote_code=True)
base_model = AutoModelForCausalLM.from_pretrained(
    base, device_map="auto", torch_dtype="auto", trust_remote_code=True
)
model = PeftModel.from_pretrained(base_model, adapters)

prompt = "You are a helpful assistant.\nUser: Summarize lottie schemas in 2 bullets.\nAssistant:"
inputs = tok(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=128)
print(tok.decode(outputs[0], skip_special_tokens=True))

Files

• —adapter_model.safetensors - LoRA adapter weights
• —adapter_config.json - PEFT configuration, includes base model id
• —eval/metrics.json - evaluation metrics and notes
• —eval/samples.md - sample generations
• —scripts/merge_adapters.py - optional: merge LoRA into full weights

Training Details

• —Method: PEFT LoRA (r=8, alpha=16, dropout=0.05; target modules: qproj,kproj,vproj,oproj)
• —Task: Causal LM instruction tuning
• —Base model: nvidia/Llama-3.1-Nemotron-8B-UltraLong-1M-Instruct
• —Hardware: [GPU type and count]; Duration: [hh:mm]
• —Seed: [seed value]; Determinism: [notes]
• —Framework versions: Transformers/PEFT/Accelerate [fill in]

Results

• —Metrics: see eval/metrics.json
• —Samples: see eval/samples.md
• —Interactive demo: https://huggingface.co/spaces/nishantmulchandani/PAMAv1-LoRA-Demo
• —Notes: [what worked, what didn't]

Quick Comparison (fill in numbers)

Limitations

• —Potential hallucinations and domain gaps
• —Long-context performance depends on prompt formatting and hardware

Responsible AI

• —No private or sensitive data included in training material
• —Be cautious for safety-sensitive or high-stakes use cases

License and Attribution

• —Base model license: see nvidia/Llama-3.1-Nemotron-8B-UltraLong-1M-Instruct on Hugging Face
• —This repo: adapters only; redistribution subject to compatibility with base model license
• —Please credit this repo for derivative research or demos

Citation

@software{pama_v1_2025,
  title = {PAMAv1 LoRA Adapters for Llama-3.1 Nemotron 8B UltraLong},
  author = {<your name or org>},
  url = {https://huggingface.co/nishantmulchandani/PAMAv1-Nemotron-8B-1M-LoRA},
  version = {v1.0.0}
}

Merge To Full Weights (Optional) Only merge and redistribute full weights if the base model license permits.

python scripts/merge_adapters.py \
  --base nvidia/Llama-3.1-Nemotron-8B-UltraLong-1M-Instruct \
  --adapters nishantmulchandani/PAMAv1-Nemotron-8B-1M-LoRA \
  --out_dir ./merged-pamav1

Changelog

• —v1.0.0: Initial release (adapters, docs, Space demo)