Felipe97/llama-cpp-compiled

1# Setting Up CUDA on Fedora2 3In this guide we setup [Nvidia CUDA](https://docs.nvidia.com/cuda/) in a toolbox container. This guide is applicable for:4 5- [Fedora Workstation](https://fedoraproject.org/workstation/)6- [Atomic Desktops for Fedora](https://fedoraproject.org/atomic-desktops/)7- [Fedora Spins](https://fedoraproject.org/spins)8- [Other Distributions](https://containertoolbx.org/distros/), including `Red Hat Enterprise Linux >= 8.5`, `Arch Linux`, and `Ubuntu`.9 10## Table of Contents11 12- [Prerequisites](#prerequisites)13- [Using the Fedora 41 CUDA Repository](#using-the-fedora-41-cuda-repository)14- [Creating a Fedora Toolbox Environment](#creating-a-fedora-toolbox-environment)15- [Installing Essential Development Tools](#installing-essential-development-tools)16- [Adding the CUDA Repository](#adding-the-cuda-repository)17- [Installing Nvidia Driver Libraries](#installing-nvidia-driver-libraries)18- [Installing the CUDA Meta-Package](#installing-the-cuda-meta-package)19- [Configuring the Environment](#configuring-the-environment)20- [Verifying the Installation](#verifying-the-installation)21- [Conclusion](#conclusion)22- [Troubleshooting](#troubleshooting)23- [Additional Notes](#additional-notes)24- [References](#references)25 26## Prerequisites27 28- **Toolbox Installed on the Host System** `Fedora Silverblue` and `Fedora Workstation` both have toolbox by default, other distributions may need to install the [toolbox package](https://containertoolbx.org/install/).29- **NVIDIA Drivers and Graphics Card installed on Host System (recommended)** To run CUDA program, such as `llama.cpp`, the host should be setup to access your NVIDIA hardware. Fedora Hosts can use the [RPM Fusion Repository](https://rpmfusion.org/Howto/NVIDIA).30- **Internet connectivity** to download packages.31 32### Using the Fedora 41 CUDA Repository33 34The latest release is 41.35 36- [Fedora 41 CUDA Repository](https://developer.download.nvidia.com/compute/cuda/repos/fedora41/x86_64/)37 38**Note:** We recommend using a toolbox environment to prevent system conflicts.39 40## Creating a Fedora Toolbox Environment41 42This guide focuses on Fedora hosts, but with small adjustments, it can work for other hosts. Using the Fedora Toolbox allows us to install the necessary packages without affecting the host system.43 44**Note:** Toolbox is available for other systems, and even without Toolbox, it is possible to use Podman or Docker.45 461. **Create a Fedora 41 Toolbox:**47 48   ```bash49   toolbox create --image registry.fedoraproject.org/fedora-toolbox:41 --container fedora-toolbox-41-cuda50   ```51 522. **Enter the Toolbox:**53 54   ```bash55   toolbox enter --container fedora-toolbox-41-cuda56   ```57 58   Inside the toolbox, you have root privileges and can install packages without affecting the host system.59 60## Installing Essential Development Tools61 621. **Synchronize the DNF Package Manager:**63 64   ```bash65   sudo dnf distro-sync66   ```67 682. **Install **Vim** the default text editor (Optional):**69 70   ```bash71   sudo dnf install vim-default-editor --allowerasing72   ```73 74   The `--allowerasing` flag will allow the removal of the conflicting `nano-default-editor` package.75 763. **Install Development Tools and Libraries:**77 78   ```bash79   sudo dnf install @c-development @development-tools cmake80   ```81 82   This installs essential packages for compiling software, including `gcc`, `make`, and other development headers.83 84## Adding the CUDA Repository85 86Add the NVIDIA CUDA repository to your DNF configuration:87 88```bash89sudo dnf config-manager addrepo --from-repofile=https://developer.download.nvidia.com/compute/cuda/repos/fedora41/x86_64/cuda-fedora41.repo90```91 92After adding the repository, synchronize the package manager again:93 94```bash95sudo dnf distro-sync96```97 98## Installing Nvidia Driver Libraries99 100First, we need to detect if the host is supplying the [NVIDIA driver libraries into the toolbox](https://github.com/containers/toolbox/blob/main/src/pkg/nvidia/nvidia.go):101 102```bash103ls -la /usr/lib64/libcuda.so.1104```105 106### If *`libcuda.so.1`* is missing:107 108```109ls: cannot access '/usr/lib64/libcuda.so.1': No such file or directory110```111 112**Explanation:**113The host dose not supply the CUDA drivers, **install them now:**114 115#### Install the Nvidia Driver Libraries on Guest:116 117```bash118sudo dnf install nvidia-driver-cuda nvidia-driver-libs nvidia-driver-cuda-libs nvidia-persistenced119```120 121### If *`libcuda.so.1`* exists:122```123lrwxrwxrwx. 1 root root 21 Mar 24 11:26 /usr/lib64/libcuda.so.1 -> libcuda.so.570.133.07124```125 126**Explanation:**127The host is supply the CUDA drivers, **we need to update the guest RPM Database accordingly:**128 129#### Update the Toolbox RPM Database to include the Host-Supplied Libraries:130 131Note: we do not actually install the libraries, we just update the DB so that the guest system knows they are supplied by the host.132 133##### 1. Download `nvidia-` parts that are supplied by the host RPM's (with dependencies)134 135```bash136sudo dnf download --destdir=/tmp/nvidia-driver-libs --resolve --arch x86_64 nvidia-driver-cuda nvidia-driver-libs nvidia-driver-cuda-libs nvidia-persistenced137```138 139##### 2. Update the RPM database to assume the installation of these packages.140 141```bash142sudo rpm --install --verbose --hash --justdb /tmp/nvidia-driver-libs/*143```144 145**Note:**146 147- The `--justdb` option only updates the RPM database, without touching the filesystem elsewhere.148 149##### Check that the RPM Database has been correctly updated:150 151**Note:** This is the same command as in the *"Install the Nvidia Driver Libraries on Guest"* for if *`libcuda.so.1`* was missing.152 153 154```bash155sudo dnf install nvidia-driver-cuda nvidia-driver-libs nvidia-driver-cuda-libs nvidia-persistenced156```157 158*(this time it will not install anything, as the database things that these packages are already installed)*159 160```161Updating and loading repositories:162Repositories loaded.163Package "nvidia-driver-cuda-3:570.124.06-1.fc41.x86_64" is already installed.164Package "nvidia-driver-libs-3:570.124.06-1.fc41.x86_64" is already installed.165Package "nvidia-driver-cuda-libs-3:570.124.06-1.fc41.x86_64" is already installed.166Package "nvidia-persistenced-3:570.124.06-1.fc41.x86_64" is already installed.167 168Nothing to do.169```170 171## Installing the CUDA Meta-Package172 173Now that the driver libraries are installed, proceed to install CUDA:174 175```bash176sudo dnf install cuda177```178 179This installs the CUDA toolkit and associated packages.180 181## Configuring the Environment182 183To use CUDA, add its binary directory to your system's `PATH`.184 1851. **Create a Profile Script:**186 187   ```bash188   sudo sh -c 'echo "export PATH=\$PATH:/usr/local/cuda/bin" >> /etc/profile.d/cuda.sh'189   ```190 191   **Explanation:**192 193   - We add to `/etc/profile.d/` as the `/etc/` folder is unique to this particular container, and is not shared with other containers or the host system.194   - The backslash `\` before `$PATH` ensures the variable is correctly written into the script.195 1962. **Make the Script Executable:**197 198   ```bash199   sudo chmod +x /etc/profile.d/cuda.sh200   ```201 2023. **Source the Script to Update Your Environment:**203 204   ```bash205   source /etc/profile.d/cuda.sh206   ```207 208   **Note:** This command updates your current shell session with the new `PATH`. The `/etc/profile.d/cuda.sh` script ensures that the CUDA binaries are available in your `PATH` for all future sessions.209 210## Verifying the Installation211 212To confirm that CUDA is correctly installed and configured, check the version of the NVIDIA CUDA Compiler (`nvcc`):213 214```bash215nvcc --version216```217 218You should see output similar to:219 220```221nvcc: NVIDIA (R) Cuda compiler driver222Copyright (c) 2005-2025 NVIDIA Corporation223Built on Fri_Feb_21_20:23:50_PST_2025224Cuda compilation tools, release 12.8, V12.8.93225Build cuda_12.8.r12.8/compiler.35583870_0226```227 228This output confirms that the CUDA compiler is accessible and indicates the installed version.229 230## Conclusion231 232You have successfully set up CUDA on Fedora within a toolbox environment using the Fedora 41 CUDA repository. By manually updating the RPM db and configuring the environment, you can develop CUDA applications without affecting your host system.233 234## Troubleshooting235 236- **Installation Failures:**237 238  - If you encounter errors during installation, carefully read the error messages. They often indicate conflicting files or missing dependencies.239  - You may use the `--excludepath` option with `rpm` to exclude conflicting files during manual RPM installations.240 241- **Rebooting the Container:**242 243  - Sometimes there may be a bug in the NVIDIA driver host passthrough (such as missing a shared library). Rebooting the container may solve this issue:244 245  ```bash246  # on the host system247  podman container restart --all248  ```249 250- **Environment Variables Not Set:**251  - If `nvcc` is not found after installation, ensure that `/usr/local/cuda/bin` is in your `PATH`.252  - Run `echo $PATH` to check if the path is included.253  - Re-source the profile script or open a new terminal session.254 255## Additional Notes256 257- **Updating CUDA in the Future:**258 259  - Keep an eye on the official NVIDIA repositories for updates to your Fedora version.260  - When an updated repository becomes available, adjust your `dnf` configuration accordingly.261 262- **Building `llama.cpp`:**263 264  - With CUDA installed, you can follow these [build instructions for `llama.cpp`](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md) to compile it with CUDA support.265  - Ensure that any CUDA-specific build flags or paths are correctly set in your build configuration.266 267- **Using the Toolbox Environment:**268  - The toolbox environment is isolated from your host system, which helps prevent conflicts.269  - Remember that system files and configurations inside the toolbox are separate from the host. By default the home directory of the user is shared between the host and the toolbox.270 271---272 273**Disclaimer:** Manually installing and modifying system packages can lead to instability of the container. The above steps are provided as a guideline and may need adjustments based on your specific system configuration. Always back up important data before making significant system changes, especially as your home folder is writable and shared with the toolbox.274 275**Acknowledgments:** Special thanks to the Fedora community and NVIDIA documentation for providing resources that assisted in creating this guide.276 277## References278 279- [Fedora Toolbox Documentation](https://docs.fedoraproject.org/en-US/fedora-silverblue/toolbox/)280- [NVIDIA CUDA Installation Guide](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html)281- [Podman Documentation](https://podman.io/get-started)282 283---284