Felipe97/llama-cpp-compiled

1# GBNF Guide2 3GBNF (GGML BNF) is a format for defining [formal grammars](https://en.wikipedia.org/wiki/Formal_grammar) to constrain model outputs in `llama.cpp`. For example, you can use it to force the model to generate valid JSON, or speak only in emojis. GBNF grammars are supported in various ways in `tools/cli`, `tools/completion` and `tools/server`.4 5## Background6 7[Backus-Naur Form (BNF)](https://en.wikipedia.org/wiki/Backus%E2%80%93Naur_form) is a notation for describing the syntax of formal languages like programming languages, file formats, and protocols. GBNF is an extension of BNF that primarily adds a few modern regex-like features.8 9## Basics10 11In GBNF, we define *production rules* that specify how a *non-terminal* (rule name) can be replaced with sequences of *terminals* (characters, specifically Unicode [code points](https://en.wikipedia.org/wiki/Code_point)) and other non-terminals. The basic format of a production rule is `nonterminal ::= sequence...`.12 13## Example14 15Before going deeper, let's look at some of the features demonstrated in `grammars/chess.gbnf`, a small chess notation grammar:16```17# `root` specifies the pattern for the overall output18root ::= (19    # it must start with the characters "1. " followed by a sequence20    # of characters that match the `move` rule, followed by a space, followed21    # by another move, and then a newline22    "1. " move " " move "\n"23 24    # it's followed by one or more subsequent moves, numbered with one or two digits25    ([1-9] [0-9]? ". " move " " move "\n")+26)27 28# `move` is an abstract representation, which can be a pawn, nonpawn, or castle.29# The `[+#]?` denotes the possibility of checking or mate signs after moves30move ::= (pawn | nonpawn | castle) [+#]?31 32pawn ::= ...33nonpawn ::= ...34castle ::= ...35```36 37## Non-Terminals and Terminals38 39Non-terminal symbols (rule names) stand for a pattern of terminals and other non-terminals. They are required to be a dashed lowercase word, like `move`, `castle`, or `check-mate`.40 41Terminals are actual characters ([code points](https://en.wikipedia.org/wiki/Code_point)). They can be specified as a sequence like `"1"` or `"O-O"` or as ranges like `[1-9]` or `[NBKQR]`.42 43## Characters and character ranges44 45Terminals support the full range of Unicode. Unicode characters can be specified directly in the grammar, for example `hiragana ::= [ぁ-ゟ]`, or with escapes: 8-bit (`\xXX`), 16-bit (`\uXXXX`) or 32-bit (`\UXXXXXXXX`).46 47Character ranges can be negated with `^`:48```49single-line ::= [^\n]+ "\n"50```51 52## Sequences and Alternatives53 54The order of symbols in a sequence matters. For example, in `"1. " move " " move "\n"`, the `"1. "` must come before the first `move`, etc.55 56Alternatives, denoted by `|`, give different sequences that are acceptable. For example, in `move ::= pawn | nonpawn | castle`, `move` can be a `pawn` move, a `nonpawn` move, or a `castle`.57 58Parentheses `()` can be used to group sequences, which allows for embedding alternatives in a larger rule or applying repetition and optional symbols (below) to a sequence.59 60## Repetition and Optional Symbols61 62- `*` after a symbol or sequence means that it can be repeated zero or more times (equivalent to `{0,}`).63- `+` denotes that the symbol or sequence should appear one or more times (equivalent to `{1,}`).64- `?` makes the preceding symbol or sequence optional (equivalent to `{0,1}`).65- `{m}` repeats the precedent symbol or sequence exactly `m` times66- `{m,}` repeats the precedent symbol or sequence at least `m` times67- `{m,n}` repeats the precedent symbol or sequence at between `m` and `n` times (included)68- `{0,n}` repeats the precedent symbol or sequence at most `n` times (included)69 70## Tokens71 72Tokens allow grammars to match specific tokenizer tokens rather than character sequences. This is useful for constraining outputs based on special tokens (like `<think>` or `</think>`).73 74Tokens can be specified in two ways:75 761. **Token ID**: Use angle brackets with the token ID in square brackets: `<[token-id]>`. For example, `<[1000]>` matches the token with ID 1000.77 782. **Token string**: Use angle brackets with the token text directly: `<token>`. For example, `<think>` will match the token whose text is exactly `<think>`. This only works if the string tokenizes to exactly one token in the vocabulary, otherwise the grammar will fail to parse.79 80You can negate token matches using the `!` prefix: `!<[1000]>` or `!<think>` matches any token *except* the specified one.81 82```83# Match a thinking block: <think>...</think>84# Using token strings (requires these to be single tokens in the vocab)85root ::= <think> thinking </think> .*86thinking ::= !</think>*87 88# Equivalent grammar using explicit token IDs89# Assumes token 1000 = <think>, token 1001 = </think>90root ::= <[1000]> thinking <[1001]> .*91thinking ::= !<[1001]>*92```93 94## Comments and newlines95 96Comments can be specified with `#`:97```98# defines optional whitespace99ws ::= [ \t\n]+100```101 102Newlines are allowed between rules and between symbols or sequences nested inside parentheses. Additionally, a newline after an alternate marker `|` will continue the current rule, even outside of parentheses.103 104## The root rule105 106In a full grammar, the `root` rule always defines the starting point of the grammar. In other words, it specifies what the entire output must match.107 108```109# a grammar for lists110root ::= ("- " item)+111item ::= [^\n]+ "\n"112```113 114## Next steps115 116This guide provides a brief overview. Check out the GBNF files in this directory (`grammars/`) for examples of full grammars. You can try them out with:117```118./llama-cli -m <model> --grammar-file grammars/some-grammar.gbnf -p 'Some prompt'119```120 121`llama.cpp` can also convert JSON schemas to grammars either ahead of time or at each request, see below.122 123## Troubleshooting124 125Grammars currently have performance gotchas (see https://github.com/ggml-org/llama.cpp/issues/4218).126 127### Efficient optional repetitions128 129A common pattern is to allow repetitions of a pattern `x` up to N times.130 131While semantically correct, the syntax `x? x? x?.... x?` (with N repetitions) may result in extremely slow sampling. Instead, you can write `x{0,N}` (or `(x (x (x ... (x)?...)?)?)?` w/ N-deep nesting in earlier llama.cpp versions).132 133## Using GBNF grammars134 135You can use GBNF grammars:136 137- In [llama-server](../tools/server)'s completion endpoints, passed as the `grammar` body field138- In [llama-cli](../tools/cli) and [llama-completion](../tools/completion), passed as the `--grammar` & `--grammar-file` flags139- With [test-gbnf-validator](../tests/test-gbnf-validator.cpp), to test them against strings.140 141## JSON Schemas → GBNF142 143`llama.cpp` supports converting a subset of https://json-schema.org/ to GBNF grammars:144 145- In [llama-server](../tools/server):146    - For any completion endpoints, passed as the `json_schema` body field147    - For the `/chat/completions` endpoint, passed inside the `response_format` body field (e.g. `{"type", "json_object", "schema": {"items": {}}}` or `{ type: "json_schema", json_schema: {"schema": ...} }`)148- In [llama-cli](../tools/cli) and [llama-completion](../tools/completion), passed as the `--json` / `-j` flag149 150> [!NOTE]151> The JSON schema is only used to constrain the model output and is not injected into the prompt. The model has no visibility into the schema, so if you want it to understand the expected structure, describe it explicitly in your prompt. This does not apply to tool calling, where schemas are injected into the prompt.152 153Take a look at [tests](../tests/test-json-schema-to-grammar.cpp) to see which features are likely supported (you'll also find usage examples in https://github.com/ggml-org/llama.cpp/pull/5978, https://github.com/ggml-org/llama.cpp/pull/6659 & https://github.com/ggml-org/llama.cpp/pull/6555).154 155```bash156llama-cli \157  -hfr bartowski/Phi-3-medium-128k-instruct-GGUF \158  -hff Phi-3-medium-128k-instruct-Q8_0.gguf \159  -j '{160    "type": "array",161    "items": {162        "type": "object",163        "properties": {164            "name": {165                "type": "string",166                "minLength": 1,167                "maxLength": 100168            },169            "age": {170                "type": "integer",171                "minimum": 0,172                "maximum": 150173            }174        },175        "required": ["name", "age"],176        "additionalProperties": false177    },178    "minItems": 10,179    "maxItems": 100180  }' \181  -p 'Generate a {name, age}[] JSON array with famous actors of all ages.'182```183 184<details>185 186<summary>Show grammar</summary>187 188The schema above converts to:189 190```191char ::= [^"\\\x7F\x00-\x1F] | [\\] (["\\bfnrt] | "u" [0-9a-fA-F]{4})192item ::= "{" space item-name-kv "," space item-age-kv "}" space193item-age ::= ([0-9] | ([1-8] [0-9] | [9] [0-9]) | "1" ([0-4] [0-9] | [5] "0")) space194item-age-kv ::= "\"age\"" space ":" space item-age195item-name ::= "\"" char{1,100} "\"" space196item-name-kv ::= "\"name\"" space ":" space item-name197root ::= "[" space item ("," space item){9,99} "]" space198space ::= | " " | "\n" [ \t]{0,20}199```200 201</details>202 203Here is also a list of known limitations (contributions welcome):204 205- `additionalProperties` defaults to `false` (produces faster grammars + reduces hallucinations).206- `"additionalProperties": true` may produce keys that contain unescaped newlines.207- Unsupported features are skipped silently. It is currently advised to use the command-line Python converter (see above) to see any warnings, and to inspect the resulting grammar / test it w/ [llama-gbnf-validator](../examples/gbnf-validator/gbnf-validator.cpp).208- Can't mix `properties` w/ `anyOf` / `oneOf` in the same type (https://github.com/ggml-org/llama.cpp/issues/7703)209- [prefixItems](https://json-schema.org/draft/2020-12/json-schema-core#name-prefixitems) is broken (but [items](https://json-schema.org/draft/2020-12/json-schema-core#name-items) works)210- `minimum`, `exclusiveMinimum`, `maximum`, `exclusiveMaximum`: only supported for `"type": "integer"` for now, not `number`211- Nested `$ref`s are broken (https://github.com/ggml-org/llama.cpp/issues/8073)212- [pattern](https://json-schema.org/draft/2020-12/json-schema-validation#name-pattern)s must start with `^` and end with `$`213- Remote `$ref`s not supported in the C++ version (Python & JavaScript versions fetch https refs)214- `string` [formats](https://json-schema.org/draft/2020-12/json-schema-validation#name-defined-formats) lack `uri`, `email`215- No [`patternProperties`](https://json-schema.org/draft/2020-12/json-schema-core#name-patternproperties)216 217And a non-exhaustive list of other unsupported features that are unlikely to be implemented (hard and/or too slow to support w/ stateless grammars):218 219- [`uniqueItems`](https://json-schema.org/draft/2020-12/json-schema-validation#name-uniqueitems)220- [`contains`](https://json-schema.org/draft/2020-12/json-schema-core#name-contains) / `minContains`221- `$anchor` (cf. [dereferencing](https://json-schema.org/draft/2020-12/json-schema-core#name-dereferencing))222- [`not`](https://json-schema.org/draft/2020-12/json-schema-core#name-not)223- [Conditionals](https://json-schema.org/draft/2020-12/json-schema-core#name-keywords-for-applying-subsche) `if` / `then` / `else` / `dependentSchemas`224 225### A word about additionalProperties226 227> [!WARNING]228> The JSON schemas spec states `object`s accept [additional properties](https://json-schema.org/understanding-json-schema/reference/object#additionalproperties) by default.229> Since this is slow and seems prone to hallucinations, we default to no additional properties.230> You can set `"additionalProperties": true` in the schema of any object to explicitly allow additional properties.231 232If you're using [Pydantic](https://pydantic.dev/) to generate schemas, you can enable additional properties with the `extra` config on each model class:233 234```python235# pip install pydantic236import json237from typing import Annotated, List238from pydantic import BaseModel, Extra, Field239class QAPair(BaseModel):240    class Config:241        extra = 'allow'  # triggers additionalProperties: true in the JSON schema242    question: str243    concise_answer: str244    justification: str245 246class Summary(BaseModel):247    class Config:248        extra = 'allow'249    key_facts: List[Annotated[str, Field(pattern='- .{5,}')]]250    question_answers: List[Annotated[List[QAPair], Field(min_items=5)]]251 252print(json.dumps(Summary.model_json_schema(), indent=2))253```254 255<details>256<summary>Show JSON schema & grammar</summary>257 258```json259{260  "$defs": {261    "QAPair": {262      "additionalProperties": true,263      "properties": {264        "question": {265          "title": "Question",266          "type": "string"267        },268        "concise_answer": {269          "title": "Concise Answer",270          "type": "string"271        },272        "justification": {273          "title": "Justification",274          "type": "string"275        }276      },277      "required": [278        "question",279        "concise_answer",280        "justification"281      ],282      "title": "QAPair",283      "type": "object"284    }285  },286  "additionalProperties": true,287  "properties": {288    "key_facts": {289      "items": {290        "pattern": "^- .{5,}$",291        "type": "string"292      },293      "title": "Key Facts",294      "type": "array"295    },296    "question_answers": {297      "items": {298        "items": {299          "$ref": "#/$defs/QAPair"300        },301        "minItems": 5,302        "type": "array"303      },304      "title": "Question Answers",305      "type": "array"306    }307  },308  "required": [309    "key_facts",310    "question_answers"311  ],312  "title": "Summary",313  "type": "object"314}315```316 317```318QAPair ::= "{" space QAPair-question-kv "," space QAPair-concise-answer-kv "," space QAPair-justification-kv ( "," space ( QAPair-additional-kv ( "," space QAPair-additional-kv )* ) )? "}" space319QAPair-additional-k ::= ["] ( [c] ([o] ([n] ([c] ([i] ([s] ([e] ([_] ([a] ([n] ([s] ([w] ([e] ([r] char+ | [^"r] char*) | [^"e] char*) | [^"w] char*) | [^"s] char*) | [^"n] char*) | [^"a] char*) | [^"_] char*) | [^"e] char*) | [^"s] char*) | [^"i] char*) | [^"c] char*) | [^"n] char*) | [^"o] char*) | [j] ([u] ([s] ([t] ([i] ([f] ([i] ([c] ([a] ([t] ([i] ([o] ([n] char+ | [^"n] char*) | [^"o] char*) | [^"i] char*) | [^"t] char*) | [^"a] char*) | [^"c] char*) | [^"i] char*) | [^"f] char*) | [^"i] char*) | [^"t] char*) | [^"s] char*) | [^"u] char*) | [q] ([u] ([e] ([s] ([t] ([i] ([o] ([n] char+ | [^"n] char*) | [^"o] char*) | [^"i] char*) | [^"t] char*) | [^"s] char*) | [^"e] char*) | [^"u] char*) | [^"cjq] char* )? ["] space320QAPair-additional-kv ::= QAPair-additional-k ":" space value321QAPair-concise-answer-kv ::= "\"concise_answer\"" space ":" space string322QAPair-justification-kv ::= "\"justification\"" space ":" space string323QAPair-question-kv ::= "\"question\"" space ":" space string324additional-k ::= ["] ( [k] ([e] ([y] ([_] ([f] ([a] ([c] ([t] ([s] char+ | [^"s] char*) | [^"t] char*) | [^"c] char*) | [^"a] char*) | [^"f] char*) | [^"_] char*) | [^"y] char*) | [^"e] char*) | [q] ([u] ([e] ([s] ([t] ([i] ([o] ([n] ([_] ([a] ([n] ([s] ([w] ([e] ([r] ([s] char+ | [^"s] char*) | [^"r] char*) | [^"e] char*) | [^"w] char*) | [^"s] char*) | [^"n] char*) | [^"a] char*) | [^"_] char*) | [^"n] char*) | [^"o] char*) | [^"i] char*) | [^"t] char*) | [^"s] char*) | [^"e] char*) | [^"u] char*) | [^"kq] char* )? ["] space325additional-kv ::= additional-k ":" space value326array ::= "[" space ( value ("," space value)* )? "]" space327boolean ::= ("true" | "false") space328char ::= [^"\\\x7F\x00-\x1F] | [\\] (["\\bfnrt] | "u" [0-9a-fA-F]{4})329decimal-part ::= [0-9]{1,16}330dot ::= [^\x0A\x0D]331integral-part ::= [0] | [1-9] [0-9]{0,15}332key-facts ::= "[" space (key-facts-item ("," space key-facts-item)*)? "]" space333key-facts-item ::= "\"" "- " key-facts-item-1{5,} "\"" space334key-facts-item-1 ::= dot335key-facts-kv ::= "\"key_facts\"" space ":" space key-facts336null ::= "null" space337number ::= ("-"? integral-part) ("." decimal-part)? ([eE] [-+]? integral-part)? space338object ::= "{" space ( string ":" space value ("," space string ":" space value)* )? "}" space339question-answers ::= "[" space (question-answers-item ("," space question-answers-item)*)? "]" space340question-answers-item ::= "[" space question-answers-item-item ("," space question-answers-item-item){4,} "]" space341question-answers-item-item ::= QAPair342question-answers-kv ::= "\"question_answers\"" space ":" space question-answers343root ::= "{" space key-facts-kv "," space question-answers-kv ( "," space ( additional-kv ( "," space additional-kv )* ) )? "}" space344space ::= | " " | "\n" [ \t]{0,20}345string ::= "\"" char* "\"" space346value ::= object | array | string | number | boolean | null347```348 349</details>350 351If you're using [Zod](https://zod.dev/), you can make your objects to explicitly allow extra properties w/ `nonstrict()` / `passthrough()` (or explicitly no extra props w/ `z.object(...).strict()` or `z.strictObject(...)`) but note that [zod-to-json-schema](https://github.com/StefanTerdell/zod-to-json-schema) currently always sets `"additionalProperties": false` anyway.352 353```js354import { z } from 'zod';355import { zodToJsonSchema } from 'zod-to-json-schema';356 357const Foo = z.object({358  age: z.number().positive(),359  email: z.string().email(),360}).strict();361 362console.log(zodToJsonSchema(Foo));363```364 365<details>366<summary>Show JSON schema & grammar</summary>367 368```json369{370  "type": "object",371  "properties": {372    "age": {373      "type": "number",374      "exclusiveMinimum": 0375    },376    "email": {377      "type": "string",378      "format": "email"379    }380  },381  "required": [382    "age",383    "email"384  ],385  "additionalProperties": false,386  "$schema": "http://json-schema.org/draft-07/schema#"387}388```389 390```391age-kv ::= "\"age\"" space ":" space number392char ::= [^"\\\x7F\x00-\x1F] | [\\] (["\\bfnrt] | "u" [0-9a-fA-F]{4})393decimal-part ::= [0-9]{1,16}394email-kv ::= "\"email\"" space ":" space string395integral-part ::= [0] | [1-9] [0-9]{0,15}396number ::= ("-"? integral-part) ("." decimal-part)? ([eE] [-+]? integral-part)? space397root ::= "{" space age-kv "," space email-kv "}" space398space ::= | " " | "\n" [ \t]{0,20}399string ::= "\"" char* "\"" space400```401 402</details>403