Felipe97/llama-cpp-compiled

1# Parsing Model Output2 3The `common` library contains a PEG parser implementation suitable for parsing4model output.5 6Types with the prefix `common_peg_*` are intended for general use and may have7applications beyond parsing model output, such as parsing user-provided regex8patterns.9 10Types with the prefix `common_chat_peg_*` are specialized helpers for model11output.12 13The parser features:14 15- Partial parsing of streaming input16- Built-in JSON parsers17- AST generation with semantics via "tagged" nodes18 19## Example20 21Below is a contrived example demonstrating how to use the PEG parser to parse22output from a model that emits arguments as JSON.23 24```cpp25auto parser = build_chat_peg_parser([&](common_chat_peg_builder & p) {26    // Build a choice of all available tools27    auto tool_choice = p.choice();28    for (const auto & tool : tools) {29        const auto & function = tool.at("function");30        std::string name = function.at("name");31        const auto   schema = common_chat_tool_parameters(function);32 33        auto tool_name = p.json_member("name", "\"" + p.literal(name) + "\"");34        auto tool_args = p.json_member("arguments", p.schema(p.json(), "tool-" + name + "-schema", schema));35 36        tool_choice |= p.rule("tool-" + name, "{" << tool_name << "," << tool_args << "}");37    }38 39    // Define the tool call structure: <tool_call>[{tool}]</tool_call>40    auto tool_call = p.trigger_rule("tool-call",41        p.sequence({42            p.literal("<tool_call>["),43            tool_choice,44            p.literal("]</tool_call>")45        })46    );47 48    // Parser accepts content, optionally followed by a tool call49    return p.sequence({50        p.content(p.until("<tool_call>")),51        p.optional(tool_call),52        p.end()53    });54});55```56 57For a more complete example, see `test_example_native()` in58[tests/test-chat-peg-parser.cpp](/tests/test-chat-peg-parser.cpp).59 60## Parsers/Combinators61 62### Basic Matchers63 64- **`eps()`** - Matches nothing and always succeeds (epsilon/empty match)65- **`start()`** - Matches the start of input (anchor `^`)66- **`end()`** - Matches the end of input (anchor `$`)67- **`literal(string)`** - Matches an exact literal string68- **`any()`** - Matches any single character (`.`)69 70### Combinators71 72- **`sequence(...)`** - Matches parsers in order; all must succeed73- **`choice(...)`** - Matches the first parser that succeeds from alternatives (ordered choice)74- **`one_or_more(p)`** - Matches one or more repetitions (`+`)75- **`zero_or_more(p)`** - Matches zero or more repetitions (`*`)76- **`optional(p)`** - Matches zero or one occurrence (`?`)77- **`repeat(p, min, max)`** - Matches between min and max repetitions (use `-1` for unbounded)78- **`repeat(p, n)`** - Matches exactly n repetitions79 80### Lookahead81 82- **`peek(p)`** - Positive lookahead: succeeds if parser succeeds without consuming input (`&`)83- **`negate(p)`** - Negative lookahead: succeeds if parser fails without consuming input (`!`)84 85### Character Classes & Utilities86 87- **`chars(classes, min, max)`** - Matches repetitions of characters from a character class88- **`space()`** - Matches zero or more whitespace characters (space, tab, newline)89- **`until(delimiter)`** - Matches characters until delimiter is found (delimiter not consumed)90- **`until_one_of(delimiters)`** - Matches characters until any delimiter in the list is found91- **`rest()`** - Matches everything remaining (`.*`)92 93### JSON Parsers94 95- **`json()`** - Complete JSON parser (objects, arrays, strings, numbers, booleans, null)96- **`json_object()`** - JSON object parser97- **`json_array()`** - JSON array parser98- **`json_string()`** - JSON string parser99- **`json_number()`** - JSON number parser100- **`json_bool()`** - JSON boolean parser101- **`json_null()`** - JSON null parser102- **`json_string_content()`** - JSON string content without surrounding quotes103- **`json_member(key, p)`** - JSON object member with specific key and value parser104 105### Grammar Building106 107- **`ref(name)`** - Creates a lightweight reference to a named rule (for recursive grammars)108- **`rule(name, p, trigger)`** - Creates a named rule and returns a reference109- **`trigger_rule(name, p)`** - Creates a trigger rule (entry point for lazy grammar generation)110- **`schema(p, name, schema, raw)`** - Wraps parser with JSON schema metadata for grammar generation111- **`schema(p, name, doc, node, raw)`** - Same, for a node of a `common_chat_schema_document` built earlier, e.g. one tool parameter112 113### AST Control114 115- **`atomic(p)`** - Prevents AST node creation for partial parses116- **`tag(tag, p)`** - Creates AST nodes with semantic tags (multiple nodes can share tags)117 118## GBNF Grammar Generation119 120The PEG parser also acts as a convenient DSL for generating GBNF grammars, with121some exceptions.122 123```cpp124data.grammar = build_grammar([&](const common_grammar_builder & builder) {125    parser.build_grammar(builder, data.grammar_lazy);126});127```128 129The notable exception is the `negate(p)` lookahead parser, which cannot be130defined as a CFG grammar and therefore does not produce a rule. Its usage131should be limited and preferably hidden behind a `schema()` parser. In many132cases, `until(delimiter)` or `until_one_of(delimiters)` is a better choice.133 134Another limitation is that the PEG parser requires an unambiguous grammar. In135contrast, the `llama-grammar` implementation can support ambiguous grammars,136though they are difficult to parse.137 138### Lazy Grammars139 140During lazy grammar generation, only rules reachable from a `trigger_rule(p)`141are emitted in the grammar. All trigger rules are added as alternations in the142root rule. It is still necessary to define trigger patterns, as the parser has143no interaction with the grammar sampling.144 145### JSON Schema146 147The `schema(p, name, schema, raw)` parser will use the `json-schema-to-grammar`148implementation to generate the grammar instead of the underlying parser.149 150The `raw` option emits a grammar suitable for a raw string instead of a JSON151string. In other words, it won't be wrapped in quotes or require escaping152quotes. It only takes effect when the schema may be a string, as reported by153`common_chat_schema::may_be_string()`, otherwise the JSON grammar is used.154 155The downside is that it can potentially lead to ambiguous grammars. For156example, if a user provides the pattern `^.*$`, the following grammar may be157generated:158 159```160root ::= "<arg>" .* "</arg>"161```162 163This creates an ambiguous grammar that cannot be parsed by the PEG parser. To164help mitigate this, if `.*` is found in the pattern, the grammar from the165underlying parser will be emitted instead.166 167## Common AST Shapes for Chat Parsing168 169Most model output can be placed in one of the following categories:170 171- Content only172- Tool calling with arguments emitted as a single JSON object173- Tool calling with arguments emitted as separate entities, either XML174  (Qwen3-Coder, MiniMax M2) or pseudo-function calls (LFM2)175 176To provide broad coverage,177[`common/chat-peg-parser.h`](/common/chat-peg-parser.h) contains builders and178mappers that help create parsers and visitors/extractors for these types. They179require parsers to tag nodes to conform to an AST "shape". This normalization180makes it easy to extract information and generalize parsing.181 182### Simple183 184The `common_chat_peg_builder` builds a `simple` parser that supports185content-only models with optional reasoning.186 187- **`reasoning(p)`** - Tag node for extracting `reasoning_content`188- **`content(p)`** - Tag node for extracting `content`189 190```cpp191build_chat_peg_parser([&](common_chat_peg_parser & p) {192    return p.sequence({193        p.optional("<think>" + p.reasoning(p.until("</think>")) + "</think>"),194        p.content(p.until("<tool_call>")),195        p.end()196    });197});198```199 200Use `common_chat_peg_mapper` to extract the content. Note that this is already201done for you in `common_chat_peg_parser` when202`chat_format == COMMON_CHAT_FORMAT_PEG_SIMPLE`.203 204```cpp205auto result = parser.parse(ctx);206 207common_chat_msg msg;208auto mapper = common_chat_peg_mapper(msg);209mapper.from_ast(ctx.ast, result);210```211 212### Native213 214The `common_chat_peg_builder` builds a `native` parser suitable for215models that emit tool arguments as a direct JSON object.216 217- **`reasoning(p)`** - Tag node for `reasoning_content`218- **`content(p)`** - Tag node for `content`219- **`tool(p)`** - Tag entirety of a single tool call220- **`tool_open(p)`** - Tag start of a tool call221- **`tool_close(p)`** - Tag end of a tool call222- **`tool_id(p)`** - Tag the tool call ID (optional)223- **`tool_name(p)`** - Tag the tool name224- **`tool_args(p)`** - Tag the tool arguments225 226```cpp227build_chat_peg_parser([&](common_chat_peg_builder & p) {228    auto get_weather_tool = p.tool(p.sequence({229        p.tool_open(p.literal("{")),230        p.json_member("name", "\"" + p.tool_name(p.literal("get_weather")) + "\""),231        p.literal(","),232        p.json_member("arguments", p.tool_args(p.json())),233        p.tool_close(p.literal("}"))234    }));235 236    return p.sequence({237        p.content(p.until("<tool_call>")),238        p.literal("<tool_call>"),239        get_weather_tool,240        p.literal("</tool_call>"),241        p.end()242    });243});244```245 246### Constructed247 248The `common_chat_peg_builder` builds a `constructed` parser249suitable for models that emit tool arguments as separate entities, such as XML250tags.251 252- **`reasoning(p)`** - Tag node for `reasoning_content`253- **`content(p)`** - Tag node for `content`254- **`tool(p)`** - Tag entirety of a single tool call255- **`tool_open(p)`** - Tag start of a tool call256- **`tool_close(p)`** - Tag end of a tool call257- **`tool_name(p)`** - Tag the tool name258- **`tool_arg(p)`** - Tag a complete tool argument (name + value)259- **`tool_arg_open(p)`** - Tag start of a tool argument260- **`tool_arg_close(p)`** - Tag end of a tool argument261- **`tool_arg_name(p)`** - Tag the argument name262- **`tool_arg_string_value(p)`** - Tag string value for the argument263- **`tool_arg_json_value(p)`** - Tag JSON value for the argument264 265```cpp266build_chat_peg_parser([&](common_chat_peg_builder & p) {267    auto location_arg = p.tool_arg(268        p.tool_arg_open("<parameter name=\"" + p.tool_arg_name(p.literal("location")) + "\">"),269        p.tool_arg_string_value(p.until("</parameter>")),270        p.tool_arg_close(p.literal("</parameter>"))271    );272 273    auto get_weather_tool = p.tool(p.sequence({274        p.tool_open("<function name=\"" + p.tool_name(p.literal("get_weather")) + "\">"),275        location_arg,276        p.tool_close(p.literal("</function>"))277    }));278 279    return p.sequence({280        p.content(p.until("<tool_call>")),281        p.literal("<tool_call>"),282        get_weather_tool,283        p.literal("</tool_call>"),284        p.end()285    });286});287```288