CoolFace
Modelpublic

Arm/hy-mt2-1-8b-2bit-translation-gguf

sourceHugging Faceapache-2.0updated 2d agoView on Hugging Face
0likes86downloads
Model Card

Hy-MT2 1.8B 2-bit Translation — GGUF

This repository packages a low-bit GGUF translation model derived from `tencent/Hy-MT2-1.8B` for multilingual translation with llama.cpp. Source and destination languages are runtime-configurable and default to English-to-Chinese translation.

✨ Key Highlights

  • —Faster translation — compared with the Q8_0 baseline on an Android™ Vivo X300 using one Arm® CPU core:
  • —Prompt prefill: 1.39× faster.
  • —Decode throughput: 49.60 tokens per second, 1.98× faster.
  • —Lower memory use — 62.36% lower average memory use than the Q8_0 baseline on an Android™ Vivo X300 using one Arm® CPU core.
  • —Smaller model — 2.89× smaller than the Q8_0 baseline.
  • —Low-bit model weights — uses proposed Q20C 2-bit per-channel quantization with Q80 token embedding tensors.
  • —Edge deployment — capable of running efficiently on edge devices such as Raspberry Pi 5.
  • —Multilingual translation — translates text across Chinese, English, French, Spanish, Italian, Japanese, Arabic, Hindi, and many more languages.

📦 Model Details

Model Description

Hy-MT2 1.8B is a multilingual translation model developed by Tencent Hunyuan. This package is derived from Tencent's 2-bit GGUF release and uses a higher-precision representation for the token embeddings.

Model Sources

  • —Base model: https://huggingface.co/tencent/Hy-MT2-1.8B
  • —Source 2-bit GGUF: https://huggingface.co/tencent/Hy-MT2-1.8B-2Bit-GGUF
  • —Upstream repository: https://github.com/Tencent-Hunyuan/Hy-MT

🚀 Get Started with the Model

🔓 Compute Flow — Early Access

The inference engine for this model package is available through the Compute Flow Early Access Program.

Want to try it?

📩 Contact us at ai-early-access@arm.com to request access.

📊 Quality evaluation

Quality evaluation results will be provided in a future update.

🎯 Performance evaluation

Performance was measured under the following conditions:

  • —Target: Android™ Vivo X300 smartphone.
  • —CPU: One Arm® CPU core.
  • —Runs: 5 warmups followed by 30 measured fresh-process runs.
  • —Pause: 60 seconds between processes.
  • —Target state: The smartphone screen was kept on.

The following methodology and definitions were used:

  • —Runtime: llama.cpp with KleidiAI on CPU.
  • —Input: One fixed English-to-Chinese translation prompt, with identical manifest settings and a deterministic sampling seed for both packages.
  • —Generation limit: At most 64 generated tokens.
  • —Prompt prefill latency is the first model generation call, including prompt processing and the first generated token.
  • —Decode throughput is the rate of subsequent generated tokens and excludes the first model generation call.
  • —Average memory is the mean process RSS sampled every 0.2 seconds across measured runs.
  • —Peak memory is the maximum process high-water mark across measured runs.
MetricQ8_0 baselineQ2_0C + Q8_0 embeddingsUplift
Prompt prefill latency (p50)309.5 ms223.0 ms1.39× faster
Decode throughput (p50)25.06 tokens/s49.60 tokens/s1.98× faster
Average memory2891.52 MiB1088.24 MiB62.36% lower
Peak memory3796.88 MiB1512.55 MiB60.16% lower
Model size1820.11 MiB629.86 MiB2.89× smaller

Lower latency and memory are better, while higher decode throughput is better.

The Q8_0 baseline uses the original `tencent/Hy-MT2-1.8B-GGUF` model.

Machine-readable results are provided in:

  • —`benchmarks/hy-mt2-1-8b-2bit-translation-vivo-x300.yaml`
  • —`benchmarks/hy-mt2-1-8b-2bit-translation-vivo-x300-baseline.yaml`

🛠️ Technical Specifications

Objective

Translate input text between user-selected source and destination languages.

Runtime Architecture

Component roleFramework / format
Text tokenization and generationllama.cpp / GGUF
Translation prompt constructionManifest-configured template

Precision and Quantization

The main model weights use Q20C 2-bit per-channel quantization. Token embedding tensors use Q80 quantization.

Q2_0C is a proposed 2-bit data type introduced in llama.cpp PR #19357 that applies per-channel quantization. Its 2-bit scheme uses Stretched Elastic Quantization (SEQ), introduced by Liu et al. (2026), to quantize weights to four levels: -1.5, -0.5, +0.5, and +1.5.

Input Specification

InputDescription
Source textUTF-8 text to translate
Source languageLanguage name selected through src_lang
Destination languageLanguage name selected through dst_lang

Output Specification

The model returns translated UTF-8 text in the selected destination language.

Manifest Configuration

OptionDefaultDescription
src_langEnglishSource language name
dst_langChineseDestination language name
prompt_templateBundled templateTranslation prompt structure
system_promptBundled instructionTranslation instruction
assistant_prefixEmpty stringOptional prefix for generated output

Source and destination languages can be changed between requests without reloading the model.

Repository Contents

  • —hy_mt2_1_8b_manifest.json — model package manifest.
  • —*.gguf — quantized translation model.
  • —metadata.yaml — model and benchmark metadata.
  • —benchmarks/*.yaml — optimized and baseline benchmark reports.
  • —SHA256SUMS — model-bundle checksums for reproducibility.

🗂️ Model and Asset Origin

The packaged GGUF was derived from the official 2-bit release and requantized with Q8_0 token embedding tensors.

🔐 Checksums

SHA256SUMS was generated by recursively hashing every regular file in the model bundle, including files in subdirectories, except the generated root SHA256SUMS and paths with a dotfile component.

From the model bundle root, verify the checked-out files with:

sh
shasum -a 256 -c SHA256SUMS