CoolFace
Modelpublic

LaraAI-Labs/Lara-TRT-EDGE-LLM-Qwen3.5-9B-INT4

sourceHugging Faceapache-2.0updated 4mo agoView on Hugging Face
0likes
Model Card

Lara-TRT-EDGE-LLM-Qwen3.5-9B-INT4

🌐 English · 한국어 · 中文

INT4 TensorRT-Edge-LLM engine (Qwen3.5-9B base) for the Lara64-TRT project — built to push time-to-first-token (TTFT) to the limit for real-time voice conversation.

Overview

Base modelQwen3.5-9B
QuantizationINT4 (groupwise GEMM / INT4 MoE)
RuntimeTensorRT-Edge-LLM
Target hardwareNVIDIA Jetson AGX Orin 64GB
StackJetPack 6.2 · CUDA 12.6 · aarch64
GoalReal-time conversation, TTFT ≈ 0.8 s
UsePrimary LLM backend for the Lara voice assistant (llm.backend: trt_edge)

Contents

  • —`engines/llm/` — main INT4 LLM engine
  • —llm.engine · embedding.safetensors · tokenizer.json · tokenizer_config.json · config.json · processed_chat_template.json
  • —`engines_mtp/` — Eagle MTP speculative decoding engines for higher decode throughput
  • —eagle_base.engine · eagle_draft.engine · embedding.safetensors · tokenizer/config files

⚠️ Compatibility

TensorRT engines are hardware- and version-locked. These engines run only on Jetson AGX Orin with JetPack 6.2 / CUDA 12.6 and a matching TensorRT-Edge-LLM build. To run on other hardware or TensorRT versions, rebuild the engine from the source ONNX.

License

Apache-2.0, following the Qwen3.5-9B base model license.