Felipe97/llama-cpp-compiled

1# llama.cpp for ET2 3- [Background](#background)4- [Limitations](#limitations)5- [Build](#build)6- [Develop](#develop)7- [Roadmap](#roadmap)8 9 10## Background11 12**ET** is a llama.cpp backend targeting the fully open source manycore13RISC-V accelerator platform [ET-SOC](https://github.com/aifoundry-org/et-man).14 15 16## Limitations17 18The ET backend runs several of the major OSS models with some limitations:19 20- Only limited set of operations is supported (check [../ops.md](../ops.md)21  and [../ops/ET.csv](../ops/ET.csv)).22- Only `q8_0`, `q4_0` (and partially `fp16`, `q4_K`) quantization is supported.23- Only one llama.cpp instance can use device at the same time (current firmware24  limitation).25- Limited (but working) MoE model support26 27As a result of the above, only select models can run fully on ET-SOC28(you can actually run any model llama.cpp supports, but some/most operations29will likely fallback to CPU backend).30 31Fully supported models:32- Qwen3 models (without MoE), e.g.33  [ggml-org/Qwen3-0.6B-GGUF:q8_0](https://huggingface.co/ggml-org/Qwen3-0.6B-GGUF/blob/main/Qwen3-0.6B-Q8_0.gguf) or34  [ggml-org/Qwen3-14B-GGUF:q8_0](https://huggingface.co/ggml-org/Qwen3-14B-GGUF/blob/main/Qwen3-14B-Q8_0.gguf).35- Llama3.2 (1B/3B), e.g.36  [lmstudio-community/Llama-3.2-1B-Instruct-GGUF:q8_0](https://huggingface.co/lmstudio-community/Llama-3.2-1B-Instruct-GGUF/blob/main/Llama-3.2-1B-Instruct-Q8_0.gguf).37- SmolLM2, e.g.38  [unsloth/SmolLM2-135M-Instruct-GGUF:q8_0](https://huggingface.co/unsloth/SmolLM2-135M-Instruct-GGUF/blob/main/SmolLM2-135M-Instruct-Q8_0.gguf)39- Llama 3.1 model family.40- RWKV v7 model family.41- TinyLLaMA42 43 44## Build45 46### I. Prerequisites47 481. **Install custom RISC-V toolchain** - Follow instructions at:49   [https://github.com/aifoundry-org/riscv-gnu-toolchain/tree/et/aifoundry](https://github.com/aifoundry-org/riscv-gnu-toolchain/tree/et/aifoundry)50 512. **Install ET platform** - Follow instructions at:52   [https://github.com/aifoundry-org/et-platform](https://github.com/aifoundry-org/et-platform)53 54Both should be installed to `/opt/et` (or set `ET_TOOLCHAIN` and `ET_PLATFORM`55environment variables accordingly).56 57```sh58# Set toolchain and ET platform path (/opt/et is default)59export ET_TOOLCHAIN=/opt/et60export ET_PLATFORM=/opt/et61```62 63### II. Build llama.cpp64 65Check out llama.cpp with ET backend (this should checkout `et` branch):66 67```sh68git clone https://github.com/aifoundry-org/llama.cpp69cd llama.cpp70```71 72Build:73 74```sh75cmake -B build -DGGML_ET=ON76cmake --build build --config Release77# Optionally:78# cmake --install build79```80 81Build targeting sysemu backend instead of physical hardware:82```sh83cmake -B build -DGGML_ET=ON -DGGML_ET_SYSEMU=ON84cmake --build build --config Release85```86 87### III. Run88 89Run llama.cpp binaries as usual. (Of course, please make sure you have the90ET-SOC device installed and kernel driver loaded).91 92```sh93llama-cli -m mymodel.gguf94# or95llama-server -hf ggml-org/Qwen3-8B-GGUF:q8_096```97 98If you want to run llama.cpp binaries (e.g. `llama-cli`) inside docker99container, you should let it access device files:100 101```sh102docker run \103    --device=/dev/et0_mgmt:/dev/et0_mgmt \104    --device=/dev/et0_ops:/dev/et0_ops \105    ...106```107 108## Develop109 110Compute kernels are developed within `ggml/src/ggml-et/et-kernels` folder.111Build is performed using custom RISC-V GNU toolchain and is managed by cmake.112At the moment kernels are build as baremetal elf files, without113standard lib or any other dependencies. All the yummy parts are written114in inline assembler.115 116Most kernels are very naive with lots of low hanging fruits left:117 118> [!IMPORTANT]119> Several assembly instructions emitted by the compiler are not implemented120> in hardware and software emulation in firmware is not ready yet.121> Eventually firmware will transparently trap unimplemented instructions122> and will emulate them inside exception handler. Until then, kernel123> build process includes step that checks compiled kernels and fails if any unimplemented124> instructions are found. Problematic ones follow:125> `FDIV.PI`, `FDIVU.PI`, `FREMU.PI`, `FREM.PI`, `FDIV.S`, `FDIV.PS`, `FSQRT.S`, `FSQRT.PS`, `FRSQ.PS`, `FSIN.PS`126>  and (long cast) `FCVT.S.L`, `FCVT.S.LU`, `FCVT.L.S`, `FCVT.LU.S`127> What this means, is that for now you should avoid doing any division involving floats,128> any trigonometry or casting longs into floats.129> Some workarounds are implemented in `math_fp.h` (`et_fdiv`, `et_powf` etc) and130> long casting (presuming longs are small enough to fit into 32bits) can be131> done via `int` like `a = (float)(int)(b)`.132 133> [!TIP]134> There are some slightly higher level helpers (abstracting more135> complex instructions like tensor extension or synchronization primitives)136> inside `et_platform`, directory `et-common-libs/include/etsoc/isa/`. It was137> originally developed for firmware needs and is not included into compute138> kernel build process. Feel free to take ideas/code from there or try linking139> it in.140 141Before committing any changes to operations and/or kernels, don't forget142to update supported ops reports (instructions at `docs/ops.md`).143 144When logging is enabled (e.g. by setting `--log-file` cli param),145each compute kernel run outputs a line with146pipe-delimited key-value pairs containing kernel level performance information.147Line is prefixed with `ET_PERF`:148 149```150ET_PERF|op=MUL_MAT|kernel=mul_mat_f32_Q8_0xf32|duration_us=3112|tensor=Qcur-0|shape=[4096,2,1,1]|start_us=48437862009|end_us=48437865121|flops=67100672151ET_PERF|op=ROPE|kernel=rope_f32|duration_us=9266|tensor=Qcur-0|shape=[128,32,2,1]|start_us=48437865128|end_us=48437874394|mode=0x0|n_dims=128|freq_base=500000.00|freq_scale=1.00152```153Keys depend on the operation, but some are always present.154`flops` in this case counts effective floating point operations and not floating155point operations per second.156 157You can enable ET-SOC runtime level ET-SOC profiling by setting environment158variable `GGML_ET_PROFILE` to a path. Profiling/tracing results will be written159to `GGML_ET_PROFILE/et_runtime_trace.json` and `GGML_ET_PROFILE/kernel_map` on exit.160 161### Uberkernel162 163The in-kernel implementation of device dispatch/kernel fusion. The ET SDK has a non-trivial op-to-op gap. `Uberkernel` (name taken from the original Esperanto AI's compiler)164dispatches multiple already existing kernel implementations with device side synchronization. Due to the processor's design, there is no natural memory visibility165horizon between sub-kernel invocations. This makes uberkernel much more difficult to develop and debug. Currently Uberkerel is hidden begind the166`GGML_ET_UBERKERNEL` environment variable and is disabled by default. Setting it to 1 enables it and provides significant performance improvements but is only167validated for the LLaMA 3.2 model family and Qwen 3.5.168 169## Roadmap170 171As of writing the documentation the ET backend is capable of running most models and smaller ones at usable speed given the low power profile of the processor. We'd172address the following capabilities in the future:173 174* Enable Uberkernel for all models175* More oprtator support176* Better TTS model support177* Enable more quantization format support178