Corneille1/youtu-llm-2b-base-blindspots

Blind Spots of tencent/Youtu-LLM-2B-Base

Model Tested

Model: tencent/Youtu-LLM-2B-Base Link: https://huggingface.co/tencent/Youtu-LLM-2B-Base

This evaluation was conducted on the base (pretrained) version of the model, not an instruction-tuned variant.

Objective

The goal of this dataset is to identify systematic failure patterns ("blind spots") of the Youtu-LLM-2B-Base model through targeted probing. The evaluation focuses on arithmetic reasoning, unit conversion, logical negation, and structured output compliance.

How the Model Was Loaded (Google Colab)

!pip -q install -U transformers accelerate torch datasets huggingface_hub

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL_ID = "tencent/Youtu-LLM-2B-Base"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)

model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    torch_dtype=torch.float16,
    device_map="auto"
)

def ask_model(prompt, max_new_tokens=120):
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    with torch.no_grad():
        output = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            do_sample=False
        )
    return tokenizer.decode(output[0], skip_special_tokens=True)

Proposed Fine-Tuning Strategy

To address the identified blind spots, I would fine-tune the model on targeted supervision datasets:

• —Curated arithmetic reasoning datasets with verified numeric labels.
• —Structured unit-conversion examples emphasizing decimal precision.
• —Contrastive logical reasoning datasets highlighting negation cases.
• —Strict structured-output datasets enforcing JSON-only responses.

Dataset Assembly Plan: I would combine synthetic data generation (programmatically generated arithmetic/unit tasks) with human-verified examples to ensure label correctness.

Estimated Dataset Size: Approximately 2,000–10,000 high-quality targeted examples would likely reduce these systematic errors.