Felipe97/llama-cpp-compiled

1> [!IMPORTANT]2> This build documentation is specific only to RISC-V SpacemiT SOCs.3 4## Build llama.cpp locally (for riscv64)5 61. Prepare Toolchain For RISCV7~~~8wget https://github.com/spacemit-com/toolchain/releases/download/v1.2.4/spacemit-toolchain-linux-glibc-x86_64-v1.2.4.tar.xz9~~~10 112. Build12Below is the build script: it requires utilizing RISC-V vector instructions for acceleration. Ensure the `GGML_CPU_RISCV64_SPACEMIT` compilation option is enabled. The currently supported optimization version is `RISCV64_SPACEMIT_IME1` and `RISCV64_SPACEMIT_IME2`, corresponding to the `RISCV64_SPACEMIT_IME_SPEC` compilation option. Compiler configurations are defined in the `riscv64-spacemit-linux-gnu-gcc.cmake` file. Please ensure you have installed the RISC-V compiler and set the environment variable via `export RISCV_ROOT_PATH={your_compiler_path}`.13```bash14 15cmake -B build \16    -DCMAKE_BUILD_TYPE=Release \17    -DGGML_CPU_RISCV64_SPACEMIT=ON \18    -DGGML_CPU_REPACK=OFF \19    -DLLAMA_OPENSSL=OFF \20    -DGGML_RVV=ON \21    -DGGML_RV_ZVFH=ON \22    -DGGML_RV_ZFH=ON \23    -DGGML_RV_ZICBOP=ON \24    -DGGML_RV_ZIHINTPAUSE=ON \25    -DGGML_RV_ZBA=ON \26    -DCMAKE_TOOLCHAIN_FILE=${PWD}/cmake/riscv64-spacemit-linux-gnu-gcc.cmake \27    -DCMAKE_INSTALL_PREFIX=build/installed28 29cmake --build build --parallel $(nproc) --config Release30 31pushd build32make install33popd34```35 36## Simulation37You can use QEMU to perform emulation on non-RISC-V architectures.38 391. Download QEMU40~~~41wget https://archive.spacemit.com/spacemit-ai/qemu/jdsk-qemu-v0.0.14.tar.gz42~~~43 442. Run Simulation45After build your llama.cpp, you can run the executable file via QEMU for simulation, for example:46~~~47export QEMU_ROOT_PATH={your QEMU file path}48export RISCV_ROOT_PATH_IME1={your RISC-V compiler path}49 50${QEMU_ROOT_PATH}/bin/qemu-riscv64 -L ${RISCV_ROOT_PATH_IME1}/sysroot -cpu max,vlen=256,elen=64,vext_spec=v1.0 ${PWD}/build/bin/llama-cli -m ${PWD}/models/Qwen2.5-0.5B-Instruct-Q4_0.gguf -t 151~~~52 53## Quantization Support For Matrix54 55| Quantization Type | X60 | A100 |56| ---: | ---: | ---: |57| Q2_K |  | :heavy_check_mark: |58| Q3_K |  | :heavy_check_mark: |59| Q4_0 | :heavy_check_mark: | :heavy_check_mark: |60| Q4_1 | :heavy_check_mark: | :heavy_check_mark: |61| Q4_K | :heavy_check_mark: | :heavy_check_mark: |62| Q5_0 |  | :heavy_check_mark: |63| Q5_1 |  | :heavy_check_mark: |64| Q5_K |  | :heavy_check_mark: |65| Q6_K |  | :heavy_check_mark: |66| Q8_0 |  | :heavy_check_mark: |67 68 69## Performance70* Spacemit(R) X6071~~~72model name      : Spacemit(R) X6073isa             : rv64imafdcv_zicbom_zicboz_zicntr_zicond_zicsr_zifencei_zihintpause_zihpm_zfh_zfhmin_zca_zcd_zba_zbb_zbc_zbs_zkt_zve32f_zve32x_zve64d_zve64f_zve64x_zvfh_zvfhmin_zvkt_sscofpmf_sstc_svinval_svnapot_svpbmt74mmu             : sv3975uarch           : spacemit,x6076mvendorid       : 0x71077marchid         : 0x800000005800000178~~~79 80| model                          |       size |     params | backend    | threads | n_ubatch | fa | mmap |            test |                  t/s |81| ------------------------------ | ---------: | ---------: | ---------- | ------: | -------: | -: | ---: | --------------: | -------------------: |82| qwen35 2B Q4_1                 |   1.19 GiB |     1.88 B | CPU        |       4 |      128 |  1 |    0 |           pp128 |         10.32 ± 0.02 |83| qwen35 2B Q4_1                 |   1.19 GiB |     1.88 B | CPU        |       4 |      128 |  1 |    0 |           tg128 |          3.07 ± 0.01 |84| qwen3 0.6B Q4_0                | 358.78 MiB |   596.05 M | CPU        |       4 |      128 |  1 |    0 |           pp128 |         49.15 ± 0.25 |85| qwen3 0.6B Q4_0                | 358.78 MiB |   596.05 M | CPU        |       4 |      128 |  1 |    0 |           tg128 |         11.73 ± 0.02 |86 87 88* Spacemit(R) A10089~~~90model name      : Spacemit(R) A10091isa             : rv64imafdcvh_zicbom_zicbop_zicboz_zicntr_zicond_zicsr_zifencei_zihintntl_zihintpause_zihpm_zimop_zaamo_zalrsc_zawrs_zfa_zfh_zfhmin_zca_zcb_zcd_zcmop_zba_zbb_zbc_zbs_zkt_zvbb_zvbc_zve32f_zve32x_zve64d_zve64f_zve64x_zvfh_zvfhmin_zvkb_zvkg_zvkned_zvknha_zvknhb_zvksed_zvksh_zvkt_smaia_smstateen_ssaia_sscofpmf_sstc_svinval_svnapot_svpbmt_sdtrig92mmu             : sv3993mvendorid       : 0x71094marchid         : 0x800000004100000295mimpid          : 0x10000000d568620096hart isa        : rv64imafdcv_zicbom_zicbop_zicboz_zicntr_zicond_zicsr_zifencei_zihintntl_zihintpause_zihpm_zimop_zaamo_zalrsc_zawrs_zfa_zfh_zfhmin_zca_zcb_zcd_zcmop_zba_zbb_zbc_zbs_zkt_zvbb_zvbc_zve32f_zve32x_zve64d_zve64f_zve64x_zvfh_zvfhmin_zvkb_zvkg_zvkned_zvknha_zvknhb_zvksed_zvksh_zvkt_smaia_smstateen_ssaia_sscofpmf_sstc_svinval_svnapot_svpbmt_sdtrig97~~~98 99| model                          |       size |     params | backend    | threads | n_ubatch | fa | mmap |            test |                  t/s |100| ------------------------------ | ---------: | ---------: | ---------- | ------: | -------: | -: | ---: | --------------: | -------------------: |101| qwen3 0.6B Q4_0                | 358.78 MiB |   596.05 M | CPU        |       8 |      128 |  1 |    0 |           pp128 |        565.83 ± 0.31 |102| qwen3 0.6B Q4_0                | 358.78 MiB |   596.05 M | CPU        |       8 |      128 |  1 |    0 |           tg128 |         55.77 ± 0.02 |103| qwen3 4B Q4_0                  |   2.21 GiB |     4.02 B | CPU        |       8 |      128 |  1 |    0 |           pp128 |         79.74 ± 0.04 |104| qwen3 4B Q4_0                  |   2.21 GiB |     4.02 B | CPU        |       8 |      128 |  1 |    0 |           tg128 |         11.29 ± 0.00 |105| qwen3moe 30B.A3B Q4_0          |  16.18 GiB |    30.53 B | CPU        |       8 |      128 |  1 |    0 |           pp128 |         57.88 ± 0.31 |106| qwen3moe 30B.A3B Q4_0          |  16.18 GiB |    30.53 B | CPU        |       8 |      128 |  1 |    0 |           tg128 |         12.79 ± 0.00 |107| qwen35 2B Q4_1                 |   1.19 GiB |     1.88 B | CPU        |       8 |      128 |  1 |    0 |           pp128 |        115.23 ± 0.04 |108| qwen35 2B Q4_1                 |   1.19 GiB |     1.88 B | CPU        |       8 |      128 |  1 |    0 |           tg128 |         16.49 ± 0.01 |109| gemma4 E4B Q4_K - Medium       |   4.76 GiB |     7.52 B | CPU        |       8 |      128 |  1 |    0 |           pp128 |         21.13 ± 0.01 |110| gemma4 E4B Q4_K - Medium       |   4.76 GiB |     7.52 B | CPU        |       8 |      128 |  1 |    0 |           tg128 |          5.66 ± 0.00 |111