RichardoC/llamafile-generator

🦙 llamafile-generator

Turn any Hugging Face GGUF model into a single self-contained `.llamafile` you can run on Linux, macOS, or Windows — generated on demand and streamed straight to your browser, with no disk and no durable storage on the server.

A llamafile bundles the llamafile loader, your GGUF weights, and a .args file into one page-aligned, runnable executable. This Space builds that bundle on the fly from a Hugging Face model reference and sends it back as a file download.

How to use

1. 1.Open the Space URL in your browser. You'll see a small form.
2. 2.Paste a Hugging Face model reference (see accepted formats below).
3. 3.Pick a llamafile version (latest by default) and a mode.
4. 4.Click ⬇ Download llamafile. Your browser downloads the .llamafile (with a progress bar, since the server sends Content-Length up front).
5. 5.Make it executable and run it:

   chmod +x MyModel.llamafile
   ./MyModel.llamafile
   # or, to chat in the browser UI:
   ./MyModel.llamafile --gui

⚠️ Ephemeral: the download is generated on demand and is not stored anywhere. If the Space is asleep the link won't work until a request wakes it. The first download after a cold start also fetches the ~350 MB llamafile loader from GitHub (a few extra seconds); later downloads reuse the cached copy until the Space sleeps.

Release lookups: the llamafile version is resolved through GitHub's releases API, which allows 60 anonymous requests an hour per IP — a busy room can exhaust that. Results are cached for an hour and served stale if the API starts refusing, and when it does refuse, releases are resolved straight from the release download URLs instead, which carry no API rate limit. No GitHub token is used or needed.

Accepted model references

The model field accepts any of:

If a directory contains several quantizations, point at a specific file (or a subdirectory that contains only the one you want) — the generator won't guess which quantization you want.

Common mistake: the base model repo (e.g. Qwen/Qwen3.5-0.8B) contains .safetensors weights, not .gguf files. You need a GGUF quantized repo — typically unsloth/<model>-GGUF or bartowski/<model>-GGUF. The error message will tell you this if you forget.

Model shapes

The generator auto-detects the shape from the files at the referenced path:

• —Single — one .gguf. .args = -m /zip/<name>.gguf + ....
• —Multimodal — a main .gguf plus a mmproj*.gguf. .args adds --mmproj /zip/<mmproj>.gguf.
• —Split / sharded — *-00001-of-NNNNN.gguf … shards. All shards are embedded; -m points at shard 00001 and llama.cpp auto-loads the siblings from /zip/.

You can override the mmproj selection with the mmproj path field (none / - clears it), and add extra args (one per line) that are inserted before the trailing ....

Modes

• —auto — generate the minimal .args for the detected shape (just -m, --mmproj if needed, your extra args, and ...). This is the default and is what most users want.
• —custom — paste the full .args contents (one argument per line) into the "Custom .args" box. A trailing ... is appended automatically if you omit it. Use this to add --ctx-size, --gpu, --server, sampling defaults, etc. Example:

  -m
  /zip/model.gguf
  --ctx-size
  4096
  --temp
  0.7

Power-user / API usage

The form submits to GET /download, so the URL is shareable and works with curl / wget:

curl -L -O 'https://<this-space>.hf.space/download?model=owner/repo:Q8_0'

Preflight with JSON (no bytes streamed):

curl 'https://<this-space>.hf.space/api/validate?model=owner/repo:Q8_0&version=0.10.3'

Example response:

{
  "ok": true,
  "model": "Qwen/Qwen3-0.6B-GGUF",
  "version": "0.10.3",
  "shape": "single",
  "filename": "Qwen3-0.6B-Q8_0.llamafile",
  "total_size": 959593837,
  "total_size_human": "915.14 MiB",
  "args": "-m\n/zip/Qwen3-0.6B-Q8_0.gguf\n...\n",
  "entries": [{"name": "Qwen3-0.6B-Q8_0.gguf", "size": 639446688}, {"name": ".args", "size": 33}],
  "exec_size": 320068791
}

HEAD /download?... returns the same headers (Content-Length, Content-Disposition) without streaming the body — useful for size preflight.

Limits (v1)

There is no per-IP rate limit — everyone behind one NAT (a workshop room, an office, a conference network) shares a client IP, so a per-IP cap would lock out a whole room.

Past the concurrency cap, requests queue rather than fail: they wait up to LLAMAFILE_GEN_QUEUE_WAIT for a slot, so a roomful of people clicking Download at once is served in waves instead of being handed error pages. Only a wait longer than that returns "server is busy". LLAMAFILE_GEN_MAX_CONCURRENT=0 removes the cap entirely.

The disk cache is on by default ($TMPDIR/llamafile-gen-cache, ~50 GB of ephemeral space on a Space). It matters more than it looks: the ~350 MB llamafile executable is identical for every download and only its trailer is held in RAM, so without a cache every request re-fetches all of it from GitHub. GGUFs are cached the same way, up to LLAMAFILE_GEN_MAX_CACHE_BYTES each, so a room downloading one model pulls it from Hugging Face once. The cache is ephemeral — it disappears when the Space sleeps — and a failed cache write never interrupts a download.

Models over the size guard are rejected with a pointer to `docs/LOCAL_BUILD.md` (build the llamafile yourself with zipalign).

How it works (short version)

The output is a deterministic, page-aligned, uncompressed (stored) ZIP64 archive, so every offset is known up front from the Hugging Face file sizes. The Space streams it byte-for-byte to your browser without ever writing to disk:

1. 1.Stream the ~350 MB llamafile APE executable from GitHub releases (cached on disk after the first fetch).
2. 2.For each GGUF: stream it once from Hugging Face, computing the ZIP CRC32 on the way past and emitting it in the entry's data descriptor (the CRC can't be backfilled into a header already sent, and HF only publishes SHA-256). The central directory carries the real CRCs, so unzip -t and Python's zipfile validate the result.
3. 3.Emit the merged central directory (the exec's own ~137 embedded entries + your files) and the ZIP64 end-of-central-directory trailer.

The server only runs a Go binary — it never executes the llamafile or zipalign itself. See `DESIGN.md` for the full design.

Local build

See `docs/LOCAL_BUILD.md` to run the server locally or to build a llamafile by hand with zipalign for models above the size guard.