CHKH01/BGE-m3-ko-GGUF

BGE-m3-ko GGUF

Korean-optimized multilingual embedding model — GGUF format for llama.cpp

BGE-m3-ko is a Korean-tuned variant of BAAI/bge-m3, fine-tuned on Korean retrieval datasets. This repository provides GGUF quantized versions for use with llama.cpp and compatible runtimes (llama-cpp-python, Ollama, etc.).

Model Details

GGUF Files

Quantization Impact

Q80 (8-bit block quantization) preserves the model's quality near-identically while reducing the model size by ~45%. For embedding tasks, the quality difference between F16 and Q80 is negligible for most use cases.

Usage

llama-server (HTTP API) — 권장

참고: llama.cpp v3.x부터 llama-embedding 바이너리는 별도로 존재하지 않습니다. 임베딩 기능은 llama-server에 통합되었습니다.

# 서버 실행 (Vulkan/CUDA/CPU 백엔드 자동 선택)
llama-server -m BGE-m3-ko.Q8_0.gguf --embedding --pooling cls --port 8080

# Request embeddings via API
curl -X POST http://localhost:8080/embedding \
  -H "Content-Type: application/json" \
  -d '{"content": "대한민국의 수도는 서울입니다"}'

# curl 응답 예시: {"embedding":[0.031159,0.055377,...],"n_tokens":8}

Python (llama-cpp-python)

from llama_cpp import Llama

llm = Llama(
    model_path="./BGE-m3-ko.Q8_0.gguf",
    embedding=True,
    n_ctx=8192,
    pooling_type=2,  # 0=None 1=Mean 2=CLS 3=Last
)

emb = llm.create_embedding("대한민국의 수도는 서울입니다")
print(len(emb["data"][0]["embedding"]))  # 1024

Original PyTorch (sentence-transformers)

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("dragonkue/BGE-m3-ko")
embeddings = model.encode(["대한민국의 수도는 서울입니다"])
print(embeddings.shape)  # (1, 1024)

Evaluation (MIRACL Korean Retrieval)

Conversion Details

Converted from dragonkue/BGE-m3-ko using llama.cpp's convert_hf_to_gguf.py at b9471.

# Convert to F16
python3 convert_hf_to_gguf.py ./dragonkue/BGE-m3-ko \
    --outfile BGE-m3-ko.f16.gguf --outtype f16

# Quantize to Q8_0
llama-quantize BGE-m3-ko.f16.gguf BGE-m3-ko.Q8_0.gguf Q8_0

GGUF Metadata

• —Architecture: bert (GGUF BERT — XLM-RoBERTa mapped to BERT arch)
• —Tokenizer: t5 type (SentencePiece Unigram)
• —Pooling: CLS
• —Causal attention: False

Intended Use

This model is designed for:

• —Korean text embeddings (primarily)
• —English + multilingual embeddings (inherited from bge-m3)
• —Semantic search / retrieval
• —Text clustering and classification
• —RAG (Retrieval-Augmented Generation) pipelines

License

Apache 2.0. The original model dragonkue/BGE-m3-ko is also Apache 2.0.

BGE-m3-ko GGUF

한국어 최적화 멀티링귀얼 임베딩 모델 — llama.cpp용 GGUF 포맷

BGE-m3-ko는 BAAI/bge-m3를 한국어 검색 데이터셋에 파인튜닝한 임베딩 모델입니다. 본 저장소는 llama.cpp 및 호환 런타임(ollama, llama-cpp-python)에서 사용 가능한 GGUF 양자화 버전을 제공합니다.

GGUF 파일

사용법

llama-server (HTTP API) — 권장

참고: llama-embedding 바이너리는 별도로 존재하지 않습니다. 임베딩은 llama-server에 통합되었습니다.

# 서버 실행 (Vulkan/CUDA/CPU)
llama-server -m BGE-m3-ko.Q8_0.gguf --embedding --pooling cls --port 8080

# 임베딩 요청
curl -X POST http://localhost:8080/embedding \
  -H "Content-Type: application/json" \
  -d '{"content": "임베딩할 문장"}'

Python (llama-cpp-python)

from llama_cpp import Llama
llm = Llama(model_path="BGE-m3-ko.Q8_0.gguf", embedding=True, n_ctx=8192, pooling_type=2)
emb = llm.create_embedding("임베딩할 문장")
print(emb["data"][0]["embedding"])

라이선스

Apache 2.0