rabeya-akter/date-arithmetic-incorrect-examples

Date Arithmetic Incorrect Examples

Overview

This dataset presents incorrect predictions made by Qwen/Qwen3.5-0.8B-Base on a focused set of date-arithmetic and calendar-reasoning questions. The goal is to provide a compact, high-signal collection of failure cases that makes it easier to study where a small base language model struggles with temporal reasoning.

The examples center on tasks such as weekday identification, date offsets, counting days between dates, recurring-event scheduling, age calculation, and nth-weekday retrieval. Only incorrect model outputs are included, so every row captures a genuine mistake rather than a random sample of performance.

Notebook

The notebook used to generate, evaluate, and curate this dataset is available here:

Date Arithmetic Error Check.ipnyb

Dataset Structure

The dataset consists of a single set of incorrect examples with the following columns:

• —id: unique example identifier
• —question: the date-arithmetic question shown to the model
• —expected_output: the reference answer computed programmatically
• —model_output: the normalized model answer used for comparison
• —raw_output: the model's original decoded response
• —match: correctness flag for the normalized answer (False for all retained rows)

How I loaded the model

The model used for this dataset is Qwen/Qwen3.5-0.8B-Base. It was loaded with the Hugging Face transformers library using AutoTokenizer and AutoModelForCausalLM, with automatic device placement and an appropriate Torch dtype depending on whether GPU support was available.

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

MODEL_ID = "Qwen/Qwen3.5-0.8B-Base"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, use_fast=True)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32,
    device_map="auto",
)
model.eval()

What the model's blind spots look like

From these examples, the main blind spots are:

• —Multi-step calendar transitions, especially when the question requires moving across months or years rather than staying within a familiar local range.
• —Leap-year and boundary-date reasoning, where February 29, month endings, and year endings introduce off-by-one or invalid-date errors.
• —Weekday alignment after offsets, where the model confuses the final weekday even when the direction of the shift is understood.
• —Counting-based temporal tasks, such as the number of days between dates or the number of a given weekday in a month.
• —Recurring schedule reasoning, where the model struggles to repeatedly apply a fixed interval over several steps.
• —Format-consistent answering, where the underlying reasoning may already be weak and answer formatting adds another source of failure.

Taken together, the errors suggest that the model does not reliably maintain precise symbolic control over calendar operations, particularly when several reasoning steps must be chained together.

What kind of data should this model be fine-tuned on?

A useful fine-tuning dataset should emphasize structured temporal reasoning rather than only surface-level date expressions. In particular, it should include:

• —date-offset problems spanning short and long ranges
• —leap-year and end-of-month edge cases
• —weekday lookup and weekday-shift problems
• —counting tasks such as days-between and weekday-frequency questions
• —recurring-event and schedule-progression questions
• —multiple answer formats with clear canonical targets
• —both correct examples and carefully chosen hard negatives or near-miss cases

The most valuable data would be examples where the correct answer depends on exact intermediate reasoning, not pattern matching alone.

How I would assemble or find such a dataset

I would combine three sources:

1. 1.Programmatically generated examples covering core calendar operations with exact ground-truth answers.
2. 2.Adversarial edge cases designed around leap years, month boundaries, same-day comparisons, large offsets, and ambiguous-looking weekday transitions.
3. 3.Model-mined failures collected by running the model on broad synthetic benchmarks and retaining the cases where it produces incorrect or weakly formatted answers.

This combination would give good coverage, preserve answer reliability, and keep the training data closely aligned with the model's actual weaknesses.

How big should the fine-tuning dataset be?

For a model of this scale, a practical starting point would be a dataset in the range of a few thousand to tens of thousands of examples, depending on diversity and quality.

A reasonable progression would be:

• —2K-5K examples for an initial targeted correction pass
• —10K-30K examples for broader coverage across date-arithmetic patterns and edge cases
• —larger collections only if they continue to add genuine variety rather than repeated templates

In this setting, quality and coverage matter more than raw volume. A smaller dataset with strong edge cases, controlled variation, and reliable labels is likely to be more useful than a much larger but repetitive set of easy examples.