afalaudn/nanbeige-3b-blind-spots

license: mit task_categories:

• —question-answering language:
• —en tags:
• —evaluation
• —blind-spots prettyname: Nanbeige-3B Blind Spots Evaluation sizecategories:
• —n<1K ---

Blind Spots Evaluation: Nanbeige/Nanbeige4-3B-Base

Model Tested

• —Model name: Nanbeige4-3B-Base
• —Parameter count: 3B
• —Architecture: LlamaForCausalLM
• —Release date: 06 December 2025
• —Confirmation: This is a pure base model with no chat template applied. It requires manual completion or few-shot prompting for structured tasks.

How to Load the Model

Include this exact working code:

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_name = "Nanbeige/Nanbeige4-3B-Base"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
tokenizer.pad_token = tokenizer.eos_token
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    dtype=torch.float16,
    device_map="auto",
    trust_remote_code=True,
)

def generate(prompt, max_new_tokens=2048):
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            do_sample=False,
            pad_token_id=tokenizer.eos_token_id,
            repetition_penalty=1.1,
        )
    new_tokens = outputs[0][inputs["input_ids"].shape[1]:]
    return tokenizer.decode(new_tokens, skip_special_tokens=True)

Evaluation Platform

• —Environment: Modal.com with NVIDIA L4 GPU (24GB VRAM)
• —Settings: Greedy decoding, max_new_tokens=2048, repetition_penalty=1.1
• —Scope: 200 prompts across 10 categories (partial evaluation of 68 prompts in this version)

Interesting Finding

Unexpected <think> tags appeared in the base model's output even though it had no explicit reasoning or RLHF training in the public description. This suggests that the pre-training data might have included a significant amount of chain-of-thought data or web crawls of model outputs (like DeepSeek's outputs) which the model learned to mimic.

Dataset Structure

Blind Spots Found

Why Does the Model Fail? (Root Cause Analysis)

• —Tokenization & Context: The model likely struggles with specific relative markers in temporal logic due to how it tokens sequence dependencies.
• —Pre-training Distribution: A strong bias towards Chinese-centric data might make performance on English-specific nuances (like "NOT" items) less robust.
• —Lack of Chat-Tuning: As a base model, it defaults to completion. Without a chat template, it "completes" the task by hallucinating a whole dialogue or additional questions.

Fine-tuning Recommendations

Recommended Datasets to Fix These Errors

• —arithmetic/math: GSM8K, MATH dataset
• —logical reasoning: LogiQA, ReClor, ProofWriter
• —Indonesian language: Indonesian SQuAD, IndoNLU
• —factual: FEVER, TriviaQA

How to Assemble Such a Dataset

1. 1.Existing Benchmarks: Subsample high-quality reasoning logs from existing datasets.
2. 2.Synthetic Generation: Use LLM to generate complex "negation" prompts and verify with a separate "critic" model.
3. 3.Human Annotation: Focus on edge cases where models typically hallucinate, specifically in temporal multi-step reasoning.

Estimated Dataset Size Needed

According to the LIMA paper, 1000 carefully curated, high-quality examples can be competitive with 50K noisy examples. For this 3B model, a targeted SFT dataset of 2000-5000 examples focusing on the specific blind spots (negation, formatting) using LoRA or full fine-tuning would likely yield significant improvements.