PleIAs/Pleias-SLM-RAG

Pleias-SLM-RAG — on-device RAG API

A tiny, fully self-contained Retrieval-Augmented Generation server. It wraps a small GGUF language model (Pleias Redline, a fine-tune of Baguettotron, ~300M parameters) with LanceDB full-text search behind a minimal Flask API — and runs entirely on CPU, including on a Raspberry Pi 5.

No GPU. No external API calls. No data leaves the machine. The code, the example databases, and the model weights all live in this repo, so a single clone gives you a working system.

What it does

Ask a question → the server retrieves the most relevant passages from a local database → the model reads them and streams back a grounded answer.

• —On-device & offline — CPU-only inference via llama.cpp, no network at runtime.
• —Streaming — answers stream token-by-token as text/plain.
• —Bring your own data — a "table" is just a folder under a dataset directory; drop yours in and query it by name.
• —Two endpoints — one with retrieval, one for feeding sources directly (testing).

Model & prompt format

The bundled model is Pleias Redline (Pleias-RAG.gguf), a ~300M parameter fine-tune of Baguettotron, quantized to Q8_0 (328 MB). It is a reasoning-style RAG model: it thinks inside a <think> block, then answers with inline <ref name="...">quote</ref> citations.

Prompts are built in ChatML with the sources inlined:

<|im_start|>user
{query}

**source_1**
{source_text}

**source_2**
{source_text}
<|im_end|>
<|im_start|>assistant
<think>

Example

Given a question and two sources, the model streams back a grounded, cited answer:

Query

<|im_start|>user
Can the Great Wall be seen from space?

**source_1**
The claim it is visible from space is a myth; the wall is at most 9.1 m wide.

**source_2**
Astronaut Yang Liwei said he could not see it from orbit.
<|im_end|>
<|im_start|>assistant
<think>

Output — the model's reasoning (truncated) then the cited answer

### 1. Query decomposition
User asks: "Can the Great Wall be seen from space?" → Simple factual question about visibility from orbital perspective. Direct information retrieval task.
...
</think>

The Great Wall of China is not visible from space. The claim that the wall is
visible from orbit is considered a myth<ref name="source_1">The claim it is
visible from space is a myth; the wall is at most 9.1 m wide.</ref>. Furthermore,
astronaut Yang Liwei stated that he could not see the Great Wall from orbit<ref
name="source_2">Astronaut Yang Liwei said he could not see it from orbit.</ref>.

Deploy

Option A — Docker (recommended)

docker pull crosash/pleias_slm_rag
docker run -p 8081:8081 crosash/pleias_slm_rag

The image bundles the model and example data — pull and use. It targets ARM64 (Raspberry Pi 5, Apple Silicon). To build your own image (Windows), see the `Dockerfile`.

Option B — local Python

git lfs install                                    # the model is stored via Git LFS
git clone https://huggingface.co/PleIAs/Pleias-SLM-RAG
cd Pleias-SLM-RAG
pip install flask lancedb pandas llama-cpp-python
python -m src.main                                 # serves on http://0.0.0.0:8081

Ready when you see Model loaded successfully. Ready for requests.

API

Both endpoints stream the model's raw output as text/plain. Use curl -N to watch tokens arrive live.

POST /ask_stream — retrieve, then answer

curl -N -X POST http://localhost:8081/ask_stream \
  -H "Content-Type: application/json" \
  -d '{"query": "What is CRSV?", "table": "en"}'

POST /raw_query — no retrieval (testing)

You supply the sources directly; they are formatted into the prompt exactly as retrieved sources would be. Handy for probing the model on controlled inputs.

curl -N -X POST http://localhost:8081/raw_query \
  -H "Content-Type: application/json" \
  -d '{
        "query": "Can the Great Wall be seen from space?",
        "sources": [
          "The claim it is visible from space is a myth; the wall is at most 9.1 m wide.",
          "Astronaut Yang Liwei said he could not see it from orbit."
        ]
      }'

Bring your own data

A dataset is a directory of tables; a table is a folder inside it holding a LanceDB dataset. The server opens the table named crsv if present, otherwise the first dataset in the folder. Point the server at your own dataset with --dataset <dir> (default: data), then select a table per request with the table field. Each table needs:

1. 1.a `text` column (its contents become the source body shown to the model), and
2. 2.a full-text-search index (search uses query_type="fts").

import lancedb

db = lancedb.connect("mydataset/faq")          # dataset "mydataset", table "faq"
tbl = db.create_table("crsv", data=[
    {"text": "First source passage ...", "url": "https://...", "lang": "en"},
    {"text": "Second source passage ...", "url": "https://...", "lang": "en"},
])
tbl.create_fts_index("text")                    # required for full-text search

Run it with python -m src.main --dataset mydataset -t faq, then query {"query": "...", "table": "faq"}. Extra columns (e.g. url, lang) are carried along as source metadata. Table names are restricted to letters, digits, _ and -.

Configuration

Server flags (python -m src.main --help):

Generation defaults (in `src/inference.py`): temperature=0.1, repetition_penalty=1.0, top_p=0.95, max_new_tokens=2048, context 4096, search_limit=3 sources per query. CPU threads are set in `src/generation.py` (n_threads=4).

Data & attribution

The example database (en, fr, both) is built from the Redline project, a corpus of international-law sources on conflict-related sexual violence (CRSV). For more information, see the original dataset: <https://huggingface.co/datasets/PleIAs/BSF_Redline>

The model is by PleIAs.