Modelpublic

DATA-ADAPT/offline-embedding

sourceHugging Faceupdated 6mo agoView on Hugging Face

0likes8downloads

Model Card

offline-embedding

Offline training with embedding-based retrieved data (2.5% of full dataset).

Note: This checkpoint is from a single random seed (seed=3) and a specific training step (step 1040). Results may vary across seeds.

Details

Key	Value
Base model	`meta-llama/Llama-2-7b-hf`
Task	MMLU
Data selection	Embedding Retrieval
Data ratio	2.5%
Online	False
LoRA rank	128
LoRA alpha	512
Target modules	qproj, kproj, vproj, oproj
Seed	3
Checkpoint step	1040

Usage

python

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")
model = PeftModel.from_pretrained(base_model, "DATA-ADAPT/offline-embedding")
tokenizer = AutoTokenizer.from_pretrained("DATA-ADAPT/offline-embedding")