Dimeji12/Dimeji-Fatima-Fellowship

Evaluating Causal and Reasoning Blind Spots in Base LLMs

Model Tested

Model Name: Qwen/Qwen2.5-3B Model Link: https://huggingface.co/Qwen/Qwen2.5-3B

How the Model Was Loaded

The model was evaluated using a Google Colab instance with a free T4 GPU. To accommodate the VRAM constraints of the hardware while maintaining inference fidelity, I utilized the transformers library alongside bitsandbytes to load the model using 8-bit quantization.

Here is the exact code used to load and test the model:

# 1. Install dependencies
!pip install -U transformers accelerate bitsandbytes

# 2. Import modules
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

# 3. Define the model
model_id = "Qwen/Qwen2.5-3B" 

# 4. Set up the 8-bit quantization configuration
quantization_config = BitsAndBytesConfig(load_in_8bit=True)

# 5. Load tokenizer and model to GPU
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, 
    device_map="auto", 
    quantization_config=quantization_config
)

# 6. Inference function
def test_blind_spot(prompt):
    inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
    outputs = model.generate(**inputs, max_new_tokens=50, temperature=0.1)
    return tokenizer.decode(outputs[0], skip_special_tokens=True)

# 7. Run test
prompt = "The primary cause of inflation in a developing economy is"
print(test_blind_spot(prompt))

Proposed Fine-Tuning Strategy

1. What kind of dataset is needed? Because this is a base model rather than an instruction-tuned model, its primary directive is pure next-token prediction. This causes it to fall into formatting loops (e.g., generating multiple-choice exam structures instead of answering) and exposes severe reasoning blind spots.

To fix this, the model requires an instruction-response dataset heavily weighted toward:

• —Chain-of-Thought (CoT) & Causal Reasoning: Data that explicitly separates causation from correlation and enforces monotonic constraints in its reasoning steps (e.g., teaching the model that increased financial stability should monotonically decrease credit risk).
• —Localized Contexts: High-quality data grounded in African socio-economic realities, as the base model heavily hallucinates when presented with Nigerian cities, local nuances, or the Naira.

2. How to assemble or find the dataset? I would use a hybrid approach to assemble this dataset:

• —For Reasoning & Causal Logic: I would utilize a larger frontier model (like GPT-4o or Llama-3-70B-Instruct) within an LLM-as-a-judge framework to synthetically generate instruction-response pairs focusing on complex causal graphs and algorithmic recourse scenarios.
• —For Cultural & Localized Logic: I would manually scrape and curate high-quality conversational data from open-source African corpora (e.g., local news archives, academic papers, and financial literacy platforms) to ensure accurate representation of local entities.

3. Dataset Size Requirement: To effectively shift the reasoning behavior of a 3B parameter model without triggering catastrophic forgetting, a Supervised Fine-Tuning (SFT) dataset of approximately 10,000 to 15,000 highly curated, diverse examples is required. To further align the model against logical fallacies and repetitive loops, this should be followed by a Direct Preference Optimization (DPO) dataset of ~5,000 preference pairs. --- license: mit ---