MiniMaxAI/MiniMax-VL-01

1# MiniMax-VL-01 Function Call Guide2 3## 📖 Introduction4 5MiniMax-VL-01 model supports function calling capability, allowing the model to identify when an external function needs to be called and output function call parameters in a structured format. This document provides detailed instructions on how to use the function calling feature of MiniMax-VL-01.6 7## 🛠️ Defining Function Calls8 9### Function Structure10 11Function calls need to be defined in the `tools` field of the request body. Each function consists of:12 13```json14{15  "tools": [16    {17      "type": "function",18      "function": {19        "name": "function_name",           // Function name, required20        "description": "function_description", // Brief description of the function's purpose21        "parameters": {                    // Parameter definition in JSON Schema format22          "type": "object",                // Overall type, fixed as "object"23          "properties": {                  // Parameter property object24            "param_name": {                // Parameter name25              "description": "Parameter description",    // Description26              "type": "string|number|boolean|array|object" // Type27            }28          },29          "required": ["param1", "param2"]  // List of required parameters30        }31      }32    }33  ]34}35```36 37### Example38 39Below is a simple example of a weather query function definition:40 41```json42"tools": [43  {44    "type": "function",45    "function": {46      "name": "get_current_weather",47      "description": "Get the latest weather for a location",48      "parameters": {49        "type": "object", 50        "properties": {51          "location": {52            "type": "string", 53            "description": "A certain city, such as Beijing, Shanghai"54          }55        }, 56        "required": ["location"]57      }58    }59  }60]61```62 63### Complete Request Example64 65Below is a complete Python code example that includes function definitions:66 67```python68payload = json.dumps({69    "model": "MiniMax-VL-01",70    "messages": [71        {72            "role": "system",73            "content": "MM Intelligent Assistant is a large-scale language model developed by MiniMax and has no interfaces to call other products. MiniMax is a China technology company that has been committed to conducting research related to large models."74        },75        {76            "role": "user",77            "content": "What's the weather like in Shanghai today?"78        }79    ],80    "tools": [81        {82            "type": "function",83            "function": {84                "name": "get_current_weather",85                "description": "Get the latest weather for a location",86                "parameters": {87                  "type": "object", 88                  "properties": {89                    "location": {90                      "type": "string", 91                      "description": "A certain city, such as Beijing, Shanghai"92                    }93                  }, 94                  "required": ["location"]95                }96            }97        }98    ],99    "tool_choice": "auto",100    "stream": True,101    "max_tokens": 10000,102    "temperature": 0.9,103    "top_p": 1104})105```106 107## 🔄 Function Call Input Format108 109When processed internally by the model, function definitions are converted to a special format and concatenated to the input text:110 111```112<beginning_of_sentence>system function_setting=functions113{"name": "get_current_weather", "description": "Get the latest weather for a location", "parameters": {"type": "object", "properties": {"location": {"type": "string", "description": "A certain city, such as Beijing, Shanghai"}}, "required": ["location"]}}<end_of_sentence>114```115 116Important notes:1171. Function definitions are placed after the system settings and before the conversation data1182. Function definitions are marked with `function_setting=functions`1193. Each function is defined as a JSON string1204. The area ends with `<end_of_sentence>`121 122## 📤 Model Function Call Output123 124When the model decides to call a function, it outputs the function call information in a special format:125 126````127<function_call>```typescript128functions.get_current_weather({"location": "Shanghai"})129```130````131 132"<function_call>" is a special token, followed by "functions.function_name(parameter json structure)". The parameters need to be string-matched and executed externally.133 134## 📥 Handling Function Results135 136After a function is successfully executed, the model will return output in the following format:137 138````typescript139```typescript140functions.get_current_weather({"location": "Shanghai"})141```142````143 144You can use the following regular expression method to extract the function name and parameters for subsequent processing:145 146````python147def parse_function_calls(content: str):148    """149    Parse the function call content returned by the model, extract function name and parameters150    151    Parameters:152        content: The original content string returned by the model153        154    Returns:155        A dictionary of parsed function call information, including function name and parameters156    """157    # Match typescript code block158    pattern = r"```typescript\n(.+?)?\n```"159    matches = re.finditer(pattern, content, re.DOTALL)160    161    for match in matches:162        function_code = match.group(1)163        # Extract function name and parameters164        function_match = re.search(r'functions\.(\w+)\((.+)\)', function_code)165        166        if not function_match:167            continue168            169        function_name = function_match.group(1)170        arguments_str = function_match.group(2)171        172        try:173            # Parse parameter JSON174            arguments = json.loads(arguments_str)175            print(f"Function call: {function_name}, Parameters: {arguments}")176            177            # Example: Handle weather query function178            if function_name == "get_current_weather":179                location = arguments.get("location", "Unknown location")180                # Build function execution result181                return {182                    "role": "function", 183                    "name": function_name, 184                    "text": json.dumps({185                        "location": location, 186                        "temperature": "25", 187                        "unit": "celsius", 188                        "weather": "Sunny"189                    }, ensure_ascii=False)190                }191        except json.JSONDecodeError as e:192            print(f"Parameter parsing failed: {arguments_str}, Error: {e}")193    194    return {}195````196 197After successfully parsing the function call, you should add the function execution result to the conversation history so that the model can access and utilize this information in subsequent interactions.198 199## 💻 Function Call Example with Transformers Library200 201The official MiniMax-VL-01 repository provides a complete example of function calling using the Transformers library. You can view the source code in the [MiniMaxAI/MiniMax-VL-01 huggingface repository](https://huggingface.co/MiniMaxAI/MiniMax-VL-01/blob/main/main.py).202 203The following is the key part of implementing function calls using the Transformers library:204 205```python206def get_default_tools():207    return [208        {209            "type": "function",210            "function": {211                "name": "get_current_weather",212                "description": "Get the latest weather for a location",213                "parameters": {214                    "type": "object", 215                    "properties": {216                        "location": {217                            "type": "string", 218                            "description": "A certain city, such as Beijing, Shanghai"219                        }220                    }, 221                    "required": ["location"]222                }223            }224        }225    ]226 227# Load model and tokenizer228tokenizer = AutoTokenizer.from_pretrained(model_id)229prompt = "What's the weather like in Shanghai today?"230messages = [231    {"role": "system", "content": [{"type": "text", "text": "You are a helpful assistant created by Minimax based on MiniMax-VL-01 model."}]},232    {"role": "user", "content": [{"type": "text", "text": prompt}]},233]234 235# Enable function call tools236tools = get_default_tools()237 238# Apply chat template and add tool definitions239text = tokenizer.apply_chat_template(240    messages,241    tokenize=False,242    add_generation_prompt=True,243    tools=tools244)245 246# Generate response247model_inputs = tokenizer(text, return_tensors="pt").to("cuda")248quantized_model = AutoModelForCausalLM.from_pretrained(249    model_id,250    torch_dtype="bfloat16",251    device_map=device_map,252    quantization_config=quantization_config,253    trust_remote_code=True,254    offload_buffers=True,255)256generation_config = GenerationConfig(257    max_new_tokens=20,258    eos_token_id=200020,259    use_cache=True,260)261 262# Execute generation263generated_ids = quantized_model.generate(**model_inputs, generation_config=generation_config)264response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]265```266 267### Running the Example268 269You can run the example code using the following command:270 271```bash272export SAFETENSORS_FAST_GPU=1273python main.py --quant_type int8 --world_size 8 --model_id <model_path> --enable_tools274```275 276Parameter description:277- `--quant_type`: Quantization type, options are "default" or "int8"278- `--world_size`: Number of GPUs, int8 quantization requires at least 8 GPUs279- `--model_id`: Model path280- `--enable_tools`: Enable function call feature281 282### Result Processing283As expected, you will get the following output:284 285````base286```typescript287functions.get_current_weather({"location": "Shanghai"})288```289````290 291You can use regular expressions to extract the function to call and its corresponding parameters:292 293````python294def try_parse_tool_calls(content: str):295    pattern = r"```typescript\n(.+?)?\n```"296    matches = re.finditer(pattern, content, re.DOTALL)297    298    for match in matches:299        function_code = match.group(1)300        function_match = re.search(r'functions\.(\w+)\((.+)\)', function_code)301        302        if not function_match:303            continue304            305        function_name = function_match.group(1)306        arguments_str = function_match.group(2)307        308        try:309            arguments = json.loads(arguments_str)310            print(f"tool_calls: [{{'type': 'function', 'function': {{'name': '{function_name}', 'arguments': {arguments}}}}}]")311            312            if function_name == "get_current_weather":313                location = arguments.get("location", "Unknown")314                return {"role": "function", "name": function_name, "text": f'{{"location": "{location}", "temperature": "25", "unit": "celsius", "weather": "Sun"}}'}315        except json.JSONDecodeError as e:316            print(f"Failed parse tools: {arguments_str}, Error: {e}")317    318    return {}319````320 321### Chat Template322 323MiniMax-VL-01 uses a specific chat template format to process function calls. The chat template is defined in `tokenizer_config.json`:324 325```json326"{% for message in messages %}{% if message['role'] == 'system' %}{{ '<beginning_of_sentence>system ai_setting=assistant\n' }}{% for item in message['content'] %}{% if item.type == 'image' %}<image>{% elif item.type == 'text' %}{{ item.text }}{% endif %}{% endfor %}{{ '<end_of_sentence>\n' }}{% endif %}{% if message['role'] == 'assistant' %}{{ '<beginning_of_sentence>ai name=assistant\n' }}{% for item in message['content'] %}{% if item.type == 'image' %}<image>{% elif item.type == 'text' %}{{ item.text }}{% endif %}{% endfor %}{{ '<end_of_sentence>\n' }}{% endif %}{% if message['role'] == 'user' %}{{ '<beginning_of_sentence>user name=user\n' }}{% for item in message['content'] %}{% if item.type == 'image' %}<image>{% elif item.type == 'text' %}{{ item.text }}{% endif %}{% endfor %}{{ '<end_of_sentence>\n' }}{% endif %}{% if message['role'] == 'function' %}{{ '<beginning_of_sentence>system function_response=functions\n' + '{\"name\": \"' + message['name'] + '\", \"response\": ' + message['content'][0]['text'] + '}' + '<end_of_sentence>\n'}}{% endif %}{% endfor %}{% if tools %}{% for function in tools %}{{ '<beginning_of_sentence>system function_setting=functions\n' + function | tojson + '<end_of_sentence>\n'}}{% endfor %}{% endif %}{% if add_generation_prompt %}{{ '<beginning_of_sentence>ai name=assistant\n' }}{% generation %}{% endgeneration %}{% endif %}"327```328 329## 📝 Important Notes330 3311. Function names should follow programming language naming conventions and avoid special characters3322. Parameter descriptions should be concise and help the model understand the parameter's purpose and constraints3333. The model does not guarantee that it will call a function; this depends on the user's input and the model's judgment3344. Function results should be returned in a structured format for easy processing by the model3355. The model might not call a function even if one is provided, depending on whether it determines a function call is appropriate for the given user query