axolotl-ai-co/text-vision-shieldstral-2k-test

A 2k sample dataset for testing the Shieldstral multimodal moderation format. Each sample is a fixed system prompt, a [text, image, text] user message, and a single yes/no answer.

Load in Axolotl via:

datasets:
  - path: Nanobit/text-vision-shieldstral-2k-test
    type: chat_template

Make sure to download the image via:

wget https://huggingface.co/datasets/Nanobit/text-vision-shieldstral-2k-test/resolve/main/African_elephant.jpg

Image source: https://upload.wikimedia.org/wikipedia/commons/e/ec/African_elephant.jpg

The image is a benign photo of an elephant in the wild. Queries cycle over safety questions it does not trigger (answer no) and factual questions about the scene (answer yes), so both classes are present without synthesising unsafe imagery.

SYSTEM_PROMPT below is the fixed judging instruction from the model card:

Judge whether the Document meets the requirements based on the Query and the Instruction provided. Note that the answer can only be "yes" or "no".

Each sample has the following format:

[
    {"role": "system", "content": [{"type": "text", "text": SYSTEM_PROMPT}]},
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "<Instruct>: ...\n\n<Query>: ...\n\n<Document>: "},
            {"type": "image", "path": LOCAL_IMAGE_PATH},
            {"type": "text", "text": " Spotted this one on safari last week.\n\n"},
        ],
    },
    {"role": "assistant", "content": [{"type": "text", "text": "no"}]},
]