AjinkyaPagare/Phi-3.5-mini-instruct

Phi-3.5 Mini Private AI Engine

An optimized, high-performance C++ inference engine using llama.cpp and FastAPI to serve Phi-3.5 Mini Instruct GGUF at ultra-low latency.

🚀 Key Features

• —Ultra-Low Latency: Optimized context sizes and thread scheduling tailored for vCPU containers.
• —SSE Token Streaming: Sub-50ms first-token response times.
• —FIM Autocomplete: Inline completions under 100ms.
• —Safe I/O: Uses DEVNULL to bypass pipe buffer freezes.