CoolFace
Modelpublic

mlx-community/InternVL3-8B-MLX-4bit

sourceHugging Faceotherupdated 7mo agoView on Hugging Face
0likes82downloads
Model Card

InternVL3-8B-MLX-4bit

This repository contains a 4-bit MLX quantized conversion of mlx-community/InternVL3-8B-bf16 for Apple Silicon inference.

Conversion Details

SettingValue
Source modelmlx-community/InternVL3-8B-bf16
Conversion toolmlx_vlm.convert
Quantization bits4
Group size64
Quantization modeaffine
Quant predicatenone (uniform quantization)

Conversion command used:

bash
python3 -m mlx_vlm convert \
  --hf-path "mlx-community/InternVL3-8B-bf16" \
  --mlx-path "./models/InternVL3-8B-4bit" \
  -q --q-bits 4 --q-group-size 64

Validation

TestStatus
Text generation load testpassed

Verification command:

bash
python3 -m mlx_vlm generate \
  --model "./models/InternVL3-8B-4bit" \
  --prompt "Reply with exactly: OK" \
  --max-tokens 8 --temperature 0

Observed response: OK

Usage

Install:

bash
python3 -m pip install -U mlx-vlm

Run locally from this folder:

bash
python3 -m mlx_vlm generate \
  --model "." \
  --prompt "Describe the image briefly." \
  --image path/to/image.jpg \
  --max-tokens 256 \
  --temperature 0

Run from Hugging Face after upload:

bash
python3 -m mlx_vlm generate \
  --model "mlx-community/InternVL3-8B-MLX-4bit" \
  --prompt "Describe the image briefly." \
  --image path/to/image.jpg \
  --max-tokens 256 \
  --temperature 0

Notes

  • —This conversion does not upload anything automatically.
  • —Quantization changes numerical behavior relative to bf16 weights.
  • —During local tests, mlx_vlm emitted an upstream tokenizer regex warning from the source model assets.

Links

  • —Source model: https://huggingface.co/mlx-community/InternVL3-8B-bf16
  • —MLX: https://github.com/ml-explore/mlx
  • —mlx-vlm: https://github.com/Blaizzy/mlx-vlm

License

Follows the upstream model license terms from the source repository.