CoolFace
Datasetpublic

akhanafer/fatimaFellowship2026

Model: https://huggingface.co/CohereLabs/tiny-aya-base Model Loading I loaded the model using the transformers' library pipeline helper function and kept the default model parameters provided by the model card. The only thing I would occasionally change was max_new_tokens, to adjust it so that it makes sense with the prompt I'm giving. I also chose to keep temperature low for more determinism: from transformers import pipeline def predict( input: str, max_new_tokens=50… See the full description on the dataset page: https://huggingface.co/datasets/akhanafer/fatimaFellowship2026.

sourceHugging Faceupdated 7mo agoView on Hugging Face
0likes10downloads
Dataset Card

Model: https://huggingface.co/CohereLabs/tiny-aya-base

Model Loading

I loaded the model using the transformers' library pipeline helper function and kept the default model parameters provided by the model card. The only thing I would occasionally change was max_new_tokens, to adjust it so that it makes sense with the prompt I'm giving. I also chose to keep temperature low for more determinism:

from transformers import pipeline

def predict(
  input: str,
  max_new_tokens=50,
  do_sample=True,
  temperature=0.1,
  top_p=0.9,
  repetition_penalty=1.1,
):
  output = pipe(
      input,
      max_new_tokens=max_new_tokens,
      do_sample=do_sample,
      temperature=temperature,
      top_p=top_p,
      repetition_penalty=repetition_penalty,
  )[0]['generated_text']

  return output

Model Finetuning

The blindspots I found are very diverse, so there's no single type of dataset that would fix all the problems that I found. However, they can be roughly grouped into distinct categories.

Category #1: Reasoning

Some blindspots in my dataset show the model's inability to reason properly, even in scenarios where commonsense would easily solve the problem. Blindspot #4 (car wash) is a perfect example. These failures likely arise because smaller base models often rely on surface statistical patterns rather than structured reasoning. Many open source datasets already exist that try to solve such issues. Examples include CommonsenseQA or StrategyQA.

Category #2: Language Switching, Homographs and Dialects

The model struggles when the prompt provided uses more than a single language. Some language switching is very common in societies where more than one language dominates the culture e.g. Franglais (mixing French and English in Canada) or mixing French and Arabic in levantine countries like Lebanon. It also struggles to understand different dialects of the same language (e.g. Gulf vs Levantine Arabic). One way of creating a dataset for these cases would be scraping forums where such speech is typically used. Reddit and X would be a great start, since users of these platforms often speak in informal manners where a mix of languages can be found. Creating synthetic data examples using an LLM is another option. The typical risks that come with collecting data using an LLM aren't as serious in this case, since it's a simple exercise that wouldn't be too impacted by things like model collapse or amplification of biases.

Category #3: Low Resource Languages

The model struggles with low-resource languages like Wolof or Swahili. Fixing these failures requires two distinct steps, unlike categories #1 and #2.

First, continued pretraining on raw monolingual text in the target language is needed before any fine-tuning can happen. This is because fine-tuning on task examples assumes the model already has basic linguistic competence in the language. If it doesn't, labeled examples won't help much since the model can't learn tasks in a language it barely understands.

Once basic competence is established through continued pretraining, fine-tuning on task-specific labeled examples can follow. Sourcing data for this step from the web is hard by definition, given they are low resource. As such, professional annotators who are native speakers of the target language would be the most reliable option.

Dataset Size

The size of the dataset will differ based on the type of failure you're trying to solve for.

Failures that fall into categories #1 and #2 will require anything between 1000 to 2000 data points per category. The model already understands the languages and has reasoning ability — it just needs to learn how to apply them more reliably. The examples also don't have to cover all possible pairs of languages, nor does it need to be trained to reason across every language it supports. A few thousand examples per category should be enough for it to generalize.

Failures that fall into category #3 require thinking about data needs in two stages. Continued pretraining requires millions of tokens of raw monolingual text to build foundational language competence — this is the bottleneck, since such text is genuinely scarce for languages like Wolof. Fine-tuning that follows would need a comparatively modest labeled dataset of a few thousand task-specific examples. The data requirements here are therefore less about volume of labeled examples and more about finding or creating sufficient raw text for the pretraining stage.