pythonplayer123/Phi-3.5-mini-instruct-Q4_K_S-GGUF

1---2license: mit3license_link: https://huggingface.co/microsoft/Phi-3.5-mini-instruct/resolve/main/LICENSE4language:5- multilingual6pipeline_tag: text-generation7tags:8- nlp9- code10- llama-cpp11- gguf-my-repo12widget:13- messages:14  - role: user15    content: Can you provide ways to eat combinations of bananas and dragonfruits?16library_name: transformers17base_model: microsoft/Phi-3.5-mini-instruct18---19 20# pythonplayer123/Phi-3.5-mini-instruct-Q4_K_S-GGUF21This model was converted to GGUF format from [`microsoft/Phi-3.5-mini-instruct`](https://huggingface.co/microsoft/Phi-3.5-mini-instruct) using llama.cpp via the ggml.ai's [GGUF-my-repo](https://huggingface.co/spaces/ggml-org/gguf-my-repo) space.22Refer to the [original model card](https://huggingface.co/microsoft/Phi-3.5-mini-instruct) for more details on the model.23 24## Use with llama.cpp25Install llama.cpp through brew (works on Mac and Linux)26 27```bash28brew install llama.cpp29 30```31Invoke the llama.cpp server or the CLI.32 33### CLI:34```bash35llama-cli --hf-repo pythonplayer123/Phi-3.5-mini-instruct-Q4_K_S-GGUF --hf-file phi-3.5-mini-instruct-q4_k_s.gguf -p "The meaning to life and the universe is"36```37 38### Server:39```bash40llama-server --hf-repo pythonplayer123/Phi-3.5-mini-instruct-Q4_K_S-GGUF --hf-file phi-3.5-mini-instruct-q4_k_s.gguf -c 204841```42 43Note: You can also use this checkpoint directly through the [usage steps](https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#usage) listed in the Llama.cpp repo as well.44 45Step 1: Clone llama.cpp from GitHub.46```47git clone https://github.com/ggerganov/llama.cpp48```49 50Step 2: Move into the llama.cpp folder and build it with `LLAMA_CURL=1` flag along with other hardware-specific flags (for ex: LLAMA_CUDA=1 for Nvidia GPUs on Linux).51```52cd llama.cpp && LLAMA_CURL=1 make53```54 55Step 3: Run inference through the main binary.56```57./llama-cli --hf-repo pythonplayer123/Phi-3.5-mini-instruct-Q4_K_S-GGUF --hf-file phi-3.5-mini-instruct-q4_k_s.gguf -p "The meaning to life and the universe is"58```59or 60```61./llama-server --hf-repo pythonplayer123/Phi-3.5-mini-instruct-Q4_K_S-GGUF --hf-file phi-3.5-mini-instruct-q4_k_s.gguf -c 204862```63