lab-flair/twinicl-bench

TwinICL

38 tasks, each with 132 underlying examples rendered in eight variants: 40,128 rows in total.

Each row contains only:

• —task: a readable task name.
• —variant: the text style or image palette.
• —input_text: the text input, or null for image examples.
• —input_image: the image input, or null for text examples.
• —answer: the expected text answer.

The eight variants are lowercase/comma, lowercase/semicolon, uppercase/comma, uppercase/semicolon, and images in neutral, warm, cool, and mint palettes. Images are 896 × 896 pixels. Each task has one file per variant, with corresponding examples in the same row order.

Position numbers start at 1. Comparison tasks answer foo for matching shapes and bar for different shapes. The even/odd task answers foo for even and bar for odd. Text answers follow the variant's letter case and separator; image answers use lowercase and commas.

from datasets import load_dataset

data = load_dataset("lab-flair/twinicl-bench", split="test")

All examples are provided together in the test split. Each row is an individual example; demonstration-and-query prompts for in-context learning must be constructed separately.