Resilient-Coders/QnA-Safety-llama

My-Meta-Llama-3.1-8B-Instruct-Open-Router-23

A thin wrapper around `meta-llama/Meta-Llama-3.1-8B-Instruct` that is packaged as a Hugging Face Inference Endpoint with a custom handler. The weights are not stored in this repo — the handler downloads them from the gated base repo at startup using an HF_TOKEN secret.

Architecture

• —handler.py — defines EndpointHandler. On __init__ it loads the tokenizer and model from meta-llama/Meta-Llama-3.1-8B-Instruct in bfloat16 with device_map="auto". On every request, __call__ applies the Llama 3.1 chat template and runs model.generate.
• —requirements.txt — pins transformers==4.51.3 (the Hugging Face Inference Toolkit breaks against transformers 5.x because it imports helpers from transformers.file_utils) and adds accelerate + huggingface_hub.

Deploying this as an Inference Endpoint

1. Prerequisites

• —A Hugging Face account that has accepted the Llama 3.1 Community License at meta-llama/Meta-Llama-3.1-8B-Instruct.
• —A User Access Token (Settings → Access Tokens) with at least read permission on gated repos.

2. Create the endpoint

From the Hugging Face UI, Deploy → Inference Endpoints on this repo, then:

3. Call it

curl https://<your-endpoint>.endpoints.huggingface.cloud \
  -H "Authorization: Bearer $HF_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "inputs": {
      "messages": [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "In one sentence, what is Hugging Face?"}
      ]
    },
    "parameters": {
      "max_new_tokens": 128,
      "do_sample": true,
      "temperature": 0.7,
      "top_p": 0.9
    }
  }'

Response shape:

{ "generated_text": "Hugging Face is ..." }

Request schema

The handler accepts both {"messages": [...]} and {"inputs": {"messages": [...]}} at the top level.

Limitations

• —The handler returns only the assistant's continuation as generated_text; it does not stream tokens and does not implement the OpenAI chat-completion schema.
• —Cold start downloads ~16 GB from the base model repo, which can take 1–3 minutes on first boot.

License

Usage is governed by the Llama 3.1 Community License. You must accept it on the base model page before this endpoint will be able to download weights.