davied-he/Octen-Embedding-4B-MLX-4bit

Octen-Embedding-4B-MLX-4bit

Community MLX 4-bit conversion of Octen/Octen-Embedding-4B for Apple silicon.

This repository is not an official Octen release. The original model was developed by Octen and is distributed under the Apache License 2.0.

Model details

The upstream checkpoint contains encoder-only Qwen3Model weights without an LM head. Its weight keys were normalized from embed_tokens.* / layers.* to the standard MLX-LM model.embed_tokens.* / model.layers.* layout before running the official mlx_lm.convert command. The final repository uses the built-in mlx_lm.models.qwen3.Model and contains no custom Python model code.

For embeddings, use the built-in model's model encoder submodule, which returns hidden states shaped [batch, sequence, 2560].

Prompt format

Use the same prompt contract as the upstream Sentence Transformers model:

Query:
Instruct: Given a web search query, retrieve relevant passages that answer the query
Query:{query}

Document:
 {document}

The document prefix is one space. Query and document embeddings must use the same checkpoint and processing contract.

Usage

import mlx.core as mx
from mlx_lm import load

repo = "davied-he/Octen-Embedding-4B-MLX-4bit"
model, tokenizer = load(repo)

QUERY_PREFIX = (
    "Instruct: Given a web search query, retrieve relevant passages that answer the query\n"
    "Query:"
)


def encode(text: str, is_query: bool) -> mx.array:
    prefix = QUERY_PREFIX if is_query else " "
    input_ids = mx.array([tokenizer.encode(prefix + text)])
    hidden_states = model.model(input_ids)
    embedding = hidden_states[:, -1, :].squeeze().astype(mx.float32)
    norm = mx.sqrt(mx.sum(mx.square(embedding)))
    return embedding / mx.maximum(norm, mx.array(1e-12))


query = encode("What is semantic retrieval?", is_query=True)
document = encode(
    "Semantic retrieval finds documents by meaning rather than exact keywords.",
    is_query=False,
)

similarity = float(mx.sum(query * document))
print(query.shape)  # (2560,)
print(similarity)

estha_embedding_config.json contains the same runtime contract in a machine-readable form.

Conversion

The upstream model uses bare encoder weight names such as embed_tokens.* and layers.*. Before conversion, those keys were deterministically prefixed with model. so they match the official MLX-LM Qwen3 model layout. No tensor values were changed during this normalization step.

mlx_lm.convert \
  --hf-path Octen/Octen-Embedding-4B \
  --mlx-path Octen-Embedding-4B-MLX-4bit \
  --quantize \
  --q-group-size 64 \
  --q-bits 4

Conversion environment:

• —mlx-lm==0.31.3
• —official mlx_lm.convert
• —affine 4-bit quantization
• —4.501 effective bits per weight reported by MLX-LM
• —no custom model_file or remote code

Validation

The published artifact was validated on Apple silicon:

• —MLX-LM loads the repository directly with built-in qwen3.Model.
• —The repository contains no custom Python model files.
• —Hidden-state output shape is [batch, sequence, 2560].
• —Query and document embeddings have shape [2560].
• —Float32 L2-normalized vector norms are 1.0.
• —On three English and Traditional Chinese samples, cosine similarity between upstream BF16 and MLX 4-bit embeddings was 0.9748–0.9782.
• —A two-query bilingual retrieval smoke test returned the expected Top-1 documents.

Artifact checksum:

model.safetensors
423d1c857da56367a84bffa182a4a875c5844a54a979c0866da38072525f6aed

Limitations

• —Quantization introduces numerical differences from the upstream BF16 model.
• —This conversion exposes embeddings, not text generation.
• —Do not mix vectors from this model with vectors generated by another model, even when both models use 2560 dimensions.
• —Retrieval quality should be evaluated on your own language and domain data.

Attribution and license

The upstream Octen/Octen-Embedding-4B model and its base Qwen/Qwen3-Embedding-4B are licensed under Apache License 2.0. This conversion preserves that license and attribution. See LICENSE.

Please cite and credit the upstream Octen model when using this conversion.