RichardErkhov/abacusai_-_Smaug-Llama-3-70B-Instruct-32K-gguf

Quantization made by Richard Erkhov.

Github

Discord

Request more models

Smaug-Llama-3-70B-Instruct-32K - GGUF

• —Model creator: https://huggingface.co/abacusai/
• —Original model: https://huggingface.co/abacusai/Smaug-Llama-3-70B-Instruct-32K/

Original model description: --- license: llama3 library_name: transformers datasets:

• —aqua_rat
• —microsoft/orca-math-word-problems-200k
• —m-a-p/CodeFeedback-Filtered-Instruction model-index:
• —name: Smaug-Llama-3-70B-Instruct-32K results:
• —task: type: text-generation name: Text Generation dataset: name: IFEval (0-Shot) type: HuggingFaceH4/ifeval args: numfewshot: 0 metrics:
• —type: instlevelstrictacc and promptlevelstrictacc value: 77.61 name: strict accuracy source: url: https://huggingface.co/spaces/open-llm-leaderboard/openllmleaderboard?query=abacusai/Smaug-Llama-3-70B-Instruct-32K name: Open LLM Leaderboard
• —task: type: text-generation name: Text Generation dataset: name: BBH (3-Shot) type: BBH args: numfewshot: 3 metrics:
• —type: accnorm value: 49.07 name: normalized accuracy source: url: https://huggingface.co/spaces/open-llm-leaderboard/openllm_leaderboard?query=abacusai/Smaug-Llama-3-70B-Instruct-32K name: Open LLM Leaderboard
• —task: type: text-generation name: Text Generation dataset: name: MATH Lvl 5 (4-Shot) type: hendrycks/competitionmath args: numfew_shot: 4 metrics:
• —type: exactmatch value: 21.22 name: exact match source: url: https://huggingface.co/spaces/open-llm-leaderboard/openllm_leaderboard?query=abacusai/Smaug-Llama-3-70B-Instruct-32K name: Open LLM Leaderboard
• —task: type: text-generation name: Text Generation dataset: name: GPQA (0-shot) type: Idavidrein/gpqa args: numfewshot: 0 metrics:
• —type: accnorm value: 6.15 name: accnorm source: url: https://huggingface.co/spaces/open-llm-leaderboard/openllmleaderboard?query=abacusai/Smaug-Llama-3-70B-Instruct-32K name: Open LLM Leaderboard
• —task: type: text-generation name: Text Generation dataset: name: MuSR (0-shot) type: TAUR-Lab/MuSR args: numfewshot: 0 metrics:
• —type: accnorm value: 12.43 name: accnorm source: url: https://huggingface.co/spaces/open-llm-leaderboard/openllmleaderboard?query=abacusai/Smaug-Llama-3-70B-Instruct-32K name: Open LLM Leaderboard
• —task: type: text-generation name: Text Generation dataset: name: MMLU-PRO (5-shot) type: TIGER-Lab/MMLU-Pro config: main split: test args: numfewshot: 5 metrics:
• —type: acc value: 41.83 name: accuracy source: url: https://huggingface.co/spaces/open-llm-leaderboard/openllmleaderboard?query=abacusai/Smaug-Llama-3-70B-Instruct-32K name: Open LLM Leaderboard ---

Smaug-Llama-3-70B-Instruct-32K

Built with Meta Llama 3

This is a 32K version of Smaug-Llama-3-70B-Instruct. It uses PoSE (https://arxiv.org/abs/2309.10400) and LoRA (https://arxiv.org/abs/2106.09685) adapter transfer. More details are coming soon.

Needle-In-A-Haystack (https://github.com/jzhang38/EasyContext) heatmap:

Model Description

• —Developed by: Abacus.AI
• —License: https://llama.meta.com/llama3/license/
• —Finetuned from model: meta-llama/Meta-Llama-3-70B-Instruct.

How to use

The prompt format is unchanged from Llama 3 70B Instruct.

Use with transformers

See the snippet below for usage with Transformers:

import transformers
import torch

model_id = "abacusai/Smaug-Llama-3-70B-Instruct"

pipeline = transformers.pipeline(
    "text-generation",
    model=model_id,
    model_kwargs={"torch_dtype": torch.bfloat16},
    device_map="auto",
)

messages = [
    {"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
    {"role": "user", "content": "Who are you?"},
]

prompt = pipeline.tokenizer.apply_chat_template(
		messages, 
		tokenize=False, 
		add_generation_prompt=True
)

terminators = [
    pipeline.tokenizer.eos_token_id,
    pipeline.tokenizer.convert_tokens_to_ids("<|eot_id|>")
]

outputs = pipeline(
    prompt,
    max_new_tokens=256,
    eos_token_id=terminators,
    do_sample=True,
    temperature=0.6,
    top_p=0.9,
)
print(outputs[0]["generated_text"][len(prompt):])

Evaluation

Arena-Hard

Arena-Hard

Score vs selected others (sourced from: (https://lmsys.org/blog/2024-04-19-arena-hard/#full-leaderboard-with-gpt-4-turbo-as-judge)). GPT-4o and Gemini-1.5-pro-latest were missing from the original blob post, and we produced those numbers from a local run using the same methodology.

Note that we believe the number of tokens/verbosity of the model strongly influences the GPT-4 judge in this case, and at least partially explains the improvement in Arena-Hard score for the 32K model.

OpenLLM Leaderboard Manual Evaluation

GSM8K The GSM8K numbers quoted here are computed using a recent release of the LM Evaluation Harness. The commit used by the leaderboard has a significant issue that impacts models that tend to use : in their responses due to a bug in the stop word configuration for GSM8K. The issue is covered in more detail in this GSM8K evaluation discussion. The score for both Llama-3 and this model are significantly different when evaluated with the updated harness as the issue with stop words has been addressed.

Open LLM Leaderboard Evaluation Results

Detailed results can be found here