Miki-T/JARVIS-Mistral-Phase1b-GGUF

JARVIS-Mistral-Phase1b-GGUF

Model ID: Miki-T/JARVIS-Mistral-Phase1b-GGUF

GGUF-format versions of the JARVIS Mistral 7B Phase 1b model for use with llama.cpp and llama-cpp-python. This is the inference-ready deployment format of Miki-T/JARVIS-Mistral-Phase1b.

Files

Model Details

This model is the result of merging two LoRA adapters into base Mistral 7B:

• —Phase 1a adapter (Miki-T/JARVIS-Mistral-Phase1a) — Macedonian language foundation trained on 500k rows of Macedonian web text
• —Phase 1b adapter (Miki-T/JARVIS-Mistral-Phase1b) — Macedonian instruction following trained on 134k instruction-response pairs

The adapters were merged sequentially using merge_and_unload(), converted to GGUF via convert_hf_to_gguf.py, and quantized to Q4KM via llama-quantize.

Usage

With llama-cpp-python

from llama_cpp import Llama

model = Llama(
    model_path="mistral-7b-jarvis-phase1b.Q4_K_M.gguf",
    n_ctx=2048,
    n_gpu_layers=40,
    verbose=False,
)

prompt = "[INST] Кој е главен град на Македонија? [/INST]"
response = model(prompt, max_tokens=256, temperature=0.7)
print(response["choices"][0]["text"])
# Output: Скопје е главниот град на Македонија.

Prompt format

Uses Mistral instruct format: [INST] <<SYS>> Your system prompt here <</SYS>> User message [/INST] Assistant response

Hardware

• —Trained on: NVIDIA RTX 5070 (12GB VRAM)
• —Q4_K_M inference: ~4.1GB VRAM, runs on any GPU with 6GB+
• —f16 inference: ~14.5GB VRAM

Merge and Conversion

# Step 1 — Merge adapters
from transformers import AutoModelForCausalLM
from peft import PeftModel
import torch

model = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-v0.1",
    torch_dtype=torch.float16,
    device_map="cpu",
)
model = PeftModel.from_pretrained(model, "path/to/phase1a/adapter")
model = model.merge_and_unload()
model = PeftModel.from_pretrained(model, "path/to/phase1b/adapter")
model = model.merge_and_unload()
model.save_pretrained("merged_model/")

# Step 2 — Convert to GGUF (requires llama.cpp)
# python convert_hf_to_gguf.py merged_model/ --outfile model.f16.gguf --outtype f16

# Step 3 — Quantize
# llama-quantize.exe model.f16.gguf model.Q4_K_M.gguf Q4_K_M

Related Repositories

• —Phase 1a adapter: Miki-T/JARVIS-Mistral-Phase1a
• —Phase 1b adapter: Miki-T/JARVIS-Mistral-Phase1b
• —Project: https://github.com/MikiTrajkovski/JARVIS

License

MIT License

Author: Miki Trajkovski | GitHub | HuggingFace

Last Updated: July 3, 2026