moonlight-labs/moonlight-qwen3-4b-v7

🌙 Moonlight Qwen3-4B v7

Moonlight is a privacy-first, sovereign AI assistant fine-tuned on top of Qwen3-4B. It is designed specifically for local and edge execution on consumer laptops (RTX 2050 4GB VRAM) and Android smartphones (via GGUF).

🌟 Key Features

• —Strict Anti-Fabrication & Honesty: Refuses to invent facts, benchmark numbers, false status reports, or imaginary tool capabilities.
• —Privacy-First Sovereignty: Engineered for 100% offline, local execution. Zero telemetry, zero cloud data transfer.
• —Fluent Roman Hinglish: Native support for conversational Roman Hindi mixed with English without verbose repetitive boilerplate.
• —Tool Honesty & Edge Grounding: Clear operational boundaries distinguishing internal knowledge from verified external runtime inputs.
• —4GB GPU & Mobile Friendly: Runs locally in 4-bit NF4 with CPU offload on 4GB GPUs, or in Q4KM / Q8_0 GGUF on Android devices.

📊 Benchmark & Evaluation Results

Tested against the frozen Moonlight Generalization Holdout Benchmark (36 test items):

🚀 Quickstart Usage

Using Hugging Face Transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

model_id = "your-username/moonlight-qwen3-4b-v7"

# 4-bit Quantization (fits in 3.5 GB VRAM)
quant_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True,
)

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=quant_config,
    device_map="auto",
    torch_dtype=torch.float16,
)

system_prompt = """You are Moonlight, a privacy-first AI assistant powered by Qwen3-4B.
Be accurate, helpful, practical, and concise.
Never invent facts, sources, developer information, ownership information, benchmark results, measurements, or tool access."""

messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": "What are the core design principles of Moonlight AI?"},
]

inputs = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt").to(model.device)

outputs = model.generate(inputs, max_new_tokens=256, do_sample=False)
response = tokenizer.decode(outputs[0][inputs.shape[1]:], skip_special_tokens=True)
print(response)

📱 Mobile & Edge Deployment (GGUF)

Moonlight can be quantized and executed offline on Android using llama.cpp, Ollama, or PocketPal AI:

# Run with Ollama
ollama create moonlight -f Modelfile
ollama run moonlight

📜 Model Details

• —Model Name: Moonlight Qwen3-4B v7
• —Base Architecture: Qwen3-4B (Dense Causal LM)
• —Parameters: 4.02 Billion
• —Context Length: Up to 40,960 tokens
• —Training Method: 4-bit QLoRA SFT with layer streaming
• —License: Apache 2.0