CoolFace
Modelpublic

yuvraj17/Llama3-8B-abliterated-Spectrum-slerp

sourceHugging Faceapache-2.0updated 2y agoView on Hugging Face
0likes25downloads
Model Card

Llama3-8B-abliterated-Spectrum-slerp

Llama3-8B-abliterated-Spectrum-slerp is a merge of the following models using LazyMergekit:

Introduction for Model Merging

Model Merging, also known as model fusion, is an effective technique that merges the parameters of multiple separate models with different capabilities to build a universal model without needing access to the original training data or expensive computation. There are bunch of methods, we can use to merge the capabilities of different models (supported by mergekit) including:

<figure>

<img src="https://cdn-uploads.huggingface.co/production/uploads/66137d95e8d2cda230ddcea6/HUflk1elPEom3PevUKu.png" width="768" height="768"> <figcaption> Merge Methods supported by MergeKit <a href="//github.com/arcee-ai/mergekit?tab=readme-ov-file#merge-methods">Reference</a> </figcaption>

</figure>

For more deep-diving into different merging techniques, visit Merge Large Language Models with mergekit.

Introduction for SLERP Merging

Spherical Linear Interpolation (SLERP) is a method used to smoothly interpolate between two vectors. It maintains a constant rate of change and preserves the geometric properties of the spherical space in which the vectors reside.

SLERP is currently the most-popular merging method, preffered over traditional methods because instead of dealing with straight-lines, the interpolation occurs on the surface of a sphere, and it has achieved improved performance to very diverse task.

But SLERP is limited to combining only two models at a time, although its possible to hierarchically combine multiple models, as shown in Mistral-7B-Merge-14-v0.1.

🧩 Configuration

yaml
slices:
  - sources:
      - model: yuvraj17/Llama-3-8B-spectrum-25
        layer_range: [0, 32]
      - model: mlabonne/Meta-Llama-3.1-8B-Instruct-abliterated
        layer_range: [0, 32]
merge_method: slerp
base_model: yuvraj17/Llama-3-8B-spectrum-25
parameters:
  t:
    - filter: self_attn
      value: [0, 0.5, 0.3, 0.7, 1]
    - filter: mlp
      value: [1, 0.5, 0.7, 0.3, 0]
    - value: 0.5
dtype: bfloat16

💻 Usage

python
!pip install -qU transformers accelerate

from transformers import AutoTokenizer
import transformers
import torch

model = "yuvraj17/Llama3-8B-abliterated-Spectrum-slerp"
messages = [{"role": "user", "content": "What is a large language model?"}]

tokenizer = AutoTokenizer.from_pretrained(model)
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    torch_dtype=torch.float16,
    device_map="auto",
)

outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
print(outputs[0]["generated_text"])

🏆 Evaluation Results

Coming soon

Special thanks & Reference