Candace352/youtu-llm-2b-base-blindspots

Youtu-LLM-2B-Base Blind Spots

This dataset contains 10 failure cases collected while probing `tencent/Youtu-LLM-2B-Base`, an open base language model on Hugging Face. The model card describes it as a Base release, lists it at 1.96B parameters, and notes support for 131,072 context length.

Model tested

• —Model: tencent/Youtu-LLM-2B-Base
• —Model type: Base model
• —Parameters: 1.96B
• —Context length: 131,072

I selected this model because it fit the assignment constraints well: it is open on Hugging Face, it is a base model rather than the separate instruct version, and it is within the required 0.6B to 6B parameter range. The model card explicitly lists both Base and Instruct versions.

Release recency

The Hugging Face model page links the paper “Youtu-LLM: Unlocking the Native Agentic Potential for Lightweight Large Language Models” and shows it as published on December 31, 2025, which places the model within the required recent release window for this assignment.

How I loaded the model

I loaded the model in Google Colab using Hugging Face Transformers. The model page notes that Youtu-LLM can be used directly with Transformers>=5.1.0.

!pip -q install -U transformers accelerate datasets huggingface_hub pandas

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "tencent/Youtu-LLM-2B-Base"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto"
)
model.eval()

How I collected the data

I wrote a set of prompts designed to test several possible blind spots in a small base language model. These probes covered tasks such as arithmetic, counting, logical reasoning, negation, short-context retrieval, constrained formatting, and simple factual comparisons. For each example, I recorded:

• —the input prompt,
• —the output I expected,
• —the model’s actual output,
• —and a short note about the failure type.

I then filtered the results to keep 10 clear failures for this dataset.

What kinds of blind spots appeared?

The failures suggest that this model can struggle when answers need to be exact, short, and tightly constrained. Across the collected probes, the most visible weaknesses were:

• —exact instruction following,
• —arithmetic or counting precision,
• —negation and logical consistency,
• —and retrieval of a specific detail from a short local context.

These errors are especially interesting because the model card reports strong benchmark performance for a lightweight base model, including results on reasoning, coding, and long-context evaluations. That makes the small, targeted failures in this dataset useful for error analysis: benchmark strength does not eliminate brittle behavior on exact-answer prompts.

What dataset should the model be fine-tuned on to fix these errors?

To improve these weaknesses, I would fine-tune the model on a targeted supervised dataset focused on:

• —short instruction-following tasks with strict answer formatting,
• —arithmetic and symbolic reasoning examples,
• —negation and logical inference examples,
• —short-context retrieval tasks,
• —and prompts requiring exact concise answers.

A useful fine-tuning set should not just contain general text. It should contain many examples where the model is rewarded for being correct and precise, not merely plausible.

How I would assemble or find such a dataset

I would combine several sources:

1. 1.Synthetic data generated programmatically for arithmetic, counting, sorting, formatting, and date reasoning.
2. 2.Public benchmark subsets for logic, commonsense reasoning, code understanding, and short QA.
3. 3.Adversarial examples created manually from the model’s observed mistakes in this dataset.
4. 4.Held-out evaluation prompts written separately so I could test whether fine-tuning improves the target behaviors rather than just memorizing the training examples.

A practical workflow would be:

• —probe the model,
• —collect real failures,
• —cluster them by error type,
• —generate many similar examples for each cluster,
• —and then fine-tune on that targeted mixture.

How large should the dataset be?

For a focused behavioral repair, I would start with about 2,000 to 5,000 examples. For broader improvement across several blind-spot categories, I would expect to need around 10,000 to 50,000 examples, depending on the quality and diversity of the data.

The key point is that a smaller, carefully targeted dataset may already improve exact-answer behavior if it directly matches the model’s most common failure modes.

Dataset fields

Each row contains:

• —id
• —category
• —input
• —expected_output
• —model_output
• —notes

Purpose

This dataset is intended for error analysis and targeted model improvement. It is not meant to be a general benchmark. Instead, it captures a small set of diverse failure cases that help reveal where a lightweight base model can still behave unreliably on simple but exact tasks.

Links

• —Model tested: `tencent/Youtu-LLM-2B-Base`
• —Dataset repo: `Candace352/youtu-llm-2b-base-blindspots`