Praz314159/Model_Zoo

Annotation Army TGI Server

High-performance Text Generation Inference server for the Annotation Army system.

Features

• —Production-ready: Using HuggingFace's Text Generation Inference (TGI)
• —High throughput: Optimized for thousands of requests
• —Low latency: Flash attention and continuous batching
• —GPU accelerated: Runs on T4/A10G for fast inference

Current Model

Mistral-7B-Instruct-v0.2

• —Excellent performance/quality balance
• —Strong instruction following
• —7B parameters

API Usage

Health Check

curl https://[your-space].hf.space/health

Generate Text

curl https://[your-space].hf.space/generate \
  -X POST \
  -H 'Content-Type: application/json' \
  -d '{
    "inputs": "Your prompt here",
    "parameters": {
      "max_new_tokens": 1024,
      "temperature": 0.1,
      "do_sample": true,
      "top_p": 0.95
    }
  }'

Generate with Streaming

curl https://[your-space].hf.space/generate_stream \
  -X POST \
  -H 'Content-Type: application/json' \
  -d '{
    "inputs": "Your prompt here",
    "parameters": {
      "max_new_tokens": 1024,
      "temperature": 0.1
    }
  }'

Configuration

To use a different model, update the MODEL_ID in the Dockerfile:

ENV MODEL_ID=your-model-id-here

Recommended models for annotation tasks:

• —mistralai/Mistral-7B-Instruct-v0.2 (current)
• —meta-llama/Llama-2-7b-chat-hf (requires access)
• —google/gemma-7b-it
• —microsoft/Phi-3-medium-4k-instruct

Performance

With GPU (T4):

• —Throughput: ~100-200 tokens/second
• —First token latency: <100ms
• —Concurrent requests: 10-50 depending on batch size

Integration

This server is designed to work with the Annotation Army system for scalable LLM-based annotation.

Space Configuration

Required:

• —Hardware: GPU (T4 minimum, A10G recommended)
• —Storage: 50GB (for model caching)

The Space will automatically:

1. 1.Download and cache the model on first startup
2. 2.Start the TGI server
3. 3.Handle incoming requests with optimal batching