Modelpublic

arcee-ai/arcee-blitz-caller-beta

sourceHugging Faceapache-2.0updated 2y agoView on Hugging Face

1likes91downloads

Model Card

Arcee Blitz Caller Beta

This repository contains a beta version of the Arcee Blitz Caller, a powerful tool for automated function calling and tool selection. Below you'll find instructions on how to launch the service using vllm as well as detailed performance metrics.

Overview

Arcee Blitz Caller is built on top of the Mistral-Small-24B-Base-2501 model and is designed for efficient and accurate tool calling operations.

Base Model

—mistralai/Mistral-Small-24B-Base-2501

Quick Start

To launch the service using vllm, run the following command:

bash

vllm serve arcee-ai/Arcee-Blitz-Caller \
    --enable-auto-tool-choice \
    --tool-call-parser hermes

Performance Metrics

Non-Live Performance

Metric	Score
AST Accuracy	85.15%
Simple AST	70.08%
Multiple AST	93.50%
Parallel AST	89.50%
Parallel Multiple AST	87.50%

Live Performance

Metric	Score
Accuracy	74.19%
Simple AST	71.71%
Multiple AST	75.02%
Parallel AST	43.75%
Parallel Multiple AST	66.67%

Multi-Turn Performance

Metric	Score
Accuracy	0.25%
Base	0.50%
Miss Function	0.00%
Miss Parameter	0.00%
Long Context	0.50%
Relevance Detection	61.11%
Irrelevance Detection	77.68%

Note: The model demonstrates strong performance in single-turn interactions but currently shows limitations in multi-turn scenarios. We are actively developing new training data to enhance multi-turn capabilities.

Current Status

This is a beta release and is under active development. While the model shows promising results in many areas, users should be aware of its current limitations, particularly in multi-turn interactions.

License

This project is licensed under the Apache-2.0 License - see the LICENSE file for details.