leninangelov/basic-chat-model

basic-chat-model

basic-chat-model is an experimental 30.55M-parameter Spanish encoder-decoder Transformer based on the T5 architecture.

The model was developed in October 2024 as a small domain-specific conversational model for a school-assistance use case. It was trained to generate short Spanish responses to inputs such as greetings, school-attendance questions, student-identifier prompts, and conversational acknowledgements.

Despite using the T5 architecture, this checkpoint was trained from scratch using a custom T5 configuration. It was not fine-tuned from `google-t5/t5-small` or another pretrained T5 checkpoint.

The model should therefore be understood as a small experimental sequence-to-sequence model rather than a general-purpose pretrained language model.

Model Details

Model Description

• —Developed by: Lenin Villanueva
• —Shared by: leninangelov
• —Development period: October 2024
• —Model repository: leninangelov/basic-chat-model
• —Model type: T5-style encoder-decoder Transformer (T5ForConditionalGeneration)
• —Parameters: 30,551,040
• —Language: Spanish (es)
• —Task: Domain-specific conversational question answering / text-to-text generation
• —Weight format: Safetensors
• —Weight precision: FP32
• —License: Apache-2.0
• —Training approach: Trained from scratch
• —Pretrained base model: None

The architecture follows the text-to-text encoder-decoder design introduced by T5:

Raffel et al. (2020), Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, arXiv:1910.10683.

The model uses the Hugging Face T5ForConditionalGeneration implementation with a substantially smaller custom configuration than the standard T5 checkpoints.

Uses

Direct Use

The model was designed as an experimental Spanish chatbot for a narrow school-assistance scenario.

The training data primarily teaches the model to handle:

• —Spanish greetings and greeting responses.
• —Questions about the arrival or attendance time of a student.
• —Requests for a student's identifier when an attendance question does not provide one.
• —Inputs containing student-like numerical identifiers.
• —Short gratitude and conversational closing expressions.

For example, the expected interaction pattern is approximately:

Usuario:
Quiero saber a qué hora llegó un estudiante hoy

Modelo:
Correcto, por favor, proporcione el documento de identificación del menor

The model generates responses directly from its learned parameters. It does not query a school database or any external source of information.

Downstream Use

This checkpoint may be useful for:

• —Educational demonstrations of small encoder-decoder Transformers.
• —Experiments with small T5 architectures.
• —Analysis of training small sequence-to-sequence models from scratch.
• —Spanish domain-specific chatbot experiments.
• —Comparing small neural conversational models against deterministic, retrieval-based, or larger pretrained approaches.
• —Further fine-tuning or continued training for research purposes.

Because the training corpus is extremely small relative to modern language-model datasets, further training on a larger and more diverse corpus would be necessary for broader applications.

Out-of-Scope Use

The model is not intended for:

• —General-purpose conversational AI.
• —Open-domain question answering.
• —Reliable factual question answering outside its training domain.
• —Production school-attendance systems.
• —Retrieving real attendance information.
• —Processing real student records without an appropriate external information system.
• —High-stakes educational, administrative, legal, medical, or financial decisions.
• —Reliable multilingual use.
• —Applications requiring robust reasoning or broad world knowledge.

In particular, attendance times generated by the model should not be interpreted as actual attendance records. They are text patterns learned from the training dataset.

Training Data

The archived training dataset contains 2,384 Spanish input-output pairs stored as question/answer pairs and subsequently converted into:

input_text
target_text

The dataset is a small, highly structured school-assistance conversational corpus.

Its approximate composition is:

The attendance examples contain three eight-digit identifier values and three attendance-time values, combined with many paraphrased questions and responses.

The complete dataset contains:

• —2,384 input-output records
• —521 unique input strings
• —521 unique target strings
• —2,380 unique input-target pairs

The dataset contains substantial repetition and paraphrasing. This was useful for the original experimental objective but limits conclusions about linguistic generalization.

No separate dataset card, external dataset source, or dataset-specific license was included in the archived project. The Apache-2.0 license stated above refers to the published model repository.

Training Procedure

Tokenizer

A custom tokenizer was trained from the same project's Spanish input and target texts.

Despite a function in the original source code being named train_sentencepiece_tokenizer, the implementation actually uses the Hugging Face tokenizers library with a:

• —BPE model
• —Whitespace pre-tokenizer
• —Training corpus containing both input and target texts

The tokenizer training requested a vocabulary size of 14,000. Because of the very small corpus, the resulting tokenizer artifact contains:

• —952 core BPE vocabulary entries
• —103 added special tokens
• —1,055 addressable tokenizer IDs in total

The added tokens consist of:

• —</s> — ID 952
• —<unk> — ID 953
• —<pad> — ID 954
• —<extra_id_0> through <extra_id_99> — IDs 955–1054

Input and target sequences were tokenized with:

• —Maximum length: 128 tokens
• —Truncation: enabled
• —Padding: fixed to 128 tokens

Model Initialization

The model was initialized directly from a custom T5Config:

config = T5Config.from_json_file("config.json")
model = T5ForConditionalGeneration(config)

There is no call to:

T5ForConditionalGeneration.from_pretrained(...)

for model initialization.

Consequently, the model did not inherit pretrained weights from T5-Small or another Google T5 checkpoint.

Training Hyperparameters

The archived training code specifies:

With 2,384 examples and a batch size of 16, one epoch contains 149 training batches. Eight epochs therefore correspond to approximately:

1,192 optimizer steps

assuming the archived single-process configuration and no gradient accumulation.

Training Objective

The model performs supervised sequence-to-sequence learning:

Spanish input text
        ↓
T5 encoder
        ↓
T5 decoder
        ↓
Spanish target response

Training optimizes token prediction over the target response using the loss implemented by T5ForConditionalGeneration.

One implementation detail should be noted: targets were padded to the fixed length of 128 before being passed to the sequence-to-sequence data collator. The archived preprocessing code does not explicitly replace existing target padding IDs with -100. Consequently, padded target positions may have participated in the training loss.

This should be corrected if the model is retrained.

Evaluation

Evaluation Protocol

The original training code uses:

train_dataset=tokenized_dataset
eval_dataset=tokenized_dataset

Therefore, the same dataset was used for both training and evaluation.

No independent validation or test split is present in the archived project.

For this reason, training-time evaluation values should not be interpreted as estimates of generalization performance, and no accuracy, BLEU, ROUGE, exact-match, or other benchmark score is claimed in this model card.

A proper evaluation would require a held-out dataset containing previously unseen questions, paraphrases, identifiers, and conversational formulations.

Bias, Risks, and Limitations

This checkpoint is an experimental small model with substantial limitations.

Narrow Training Domain

The model was trained only on a very small Spanish school-assistance dataset. It did not undergo broad Spanish-language pretraining and therefore should not be expected to possess the linguistic coverage or world knowledge of pretrained T5, mT5, or modern instruction-tuned language models.

Dataset Repetition

Although there are 2,384 training records, there are only 521 unique input strings. Many training examples are repeated or paraphrased variations of the same small set of intents.

The model may therefore memorize patterns rather than learn broad conversational capabilities.

Conflicting Attendance Labels

The archived dataset intentionally or inadvertently associates identical student-identifier questions with different attendance times in different sections of the dataset.

Analysis of the original dataset found 354 exact input strings containing attendance information that are associated with more than one time label.

Consequently, the model cannot be expected to learn a deterministic relationship between an identifier and a specific attendance time.

This is particularly important because the generated attendance time is not connected to any external database.

No Independent Evaluation Set

The training dataset was also used as the evaluation dataset. Generalization to unseen inputs was therefore not measured during the original experiment.

Tokenizer / Model Vocabulary Mismatch

The T5 model is configured with:

vocab_size = 14,000

and therefore contains a shared embedding matrix of shape:

14,000 × 384

However, the final tokenizer exposes only approximately 1,055 token IDs.

This means that a large fraction of the model's 14,000-token output space does not correspond to tokens produced by the archived tokenizer. This increases parameter count unnecessarily and is another reason this checkpoint should be treated as an experimental artifact.

EOS Token Metadata Mismatch

The published model configuration stores:

eos_token_id = 1

while the tokenizer assigns:

</s> = 952

The tokenizer and model therefore disagree about the end-of-sequence token.

When using the existing checkpoint, generation code can explicitly provide the tokenizer's EOS token ID. A future corrected release should align these values in the model configuration.

Target Padding

Because target sequences were already padded to length 128 during preprocessing, padding positions may not have been masked with the standard -100 ignore index before loss calculation.

This may have encouraged the model to learn excessive padding behavior.

Factual Reliability

Generated attendance times, student identifiers, or other factual-looking responses are generated language-model outputs. They are not retrieved facts and must not be used as actual school records.

Safety

The model did not undergo dedicated safety alignment, red-teaming, toxicity evaluation, fairness evaluation, or adversarial testing.

Recommendations

This checkpoint is best treated as a historical experimental and educational artifact.

For a new version of the model, the following changes are recommended:

1. 1.Build distinct training, validation, and test splits.
2. 2.Ensure that equivalent inputs have logically consistent target labels.
3. 3.Connect attendance lookup to a deterministic database or retrieval layer instead of expecting the language model to memorize attendance values.
4. 4.Use the language model only for natural-language understanding and response formulation.
5. 5.Align the model vocabulary size with the actual tokenizer vocabulary.
6. 6.Set eos_token_id, pad_token_id, and decoder_start_token_id consistently.
7. 7.Mask target padding tokens with -100 during training.
8. 8.Evaluate on unseen Spanish paraphrases and previously unseen identifier values.
9. 9.Use a larger and more diverse Spanish corpus if broader conversational capability is desired.
10. 10.Avoid using real personal/student identifiers in experimental training data unless appropriate privacy controls are implemented.

How to Get Started with the Model

The model can be loaded as a sequence-to-sequence model using Hugging Face Transformers.

Because the archived model configuration and tokenizer contain different EOS IDs, the example below explicitly supplies the tokenizer's EOS token during generation.

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

model_id = "leninangelov/basic-chat-model"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id)

question = "Quiero saber a qué hora llegó un estudiante hoy"

inputs = tokenizer(
    question,
    return_tensors="pt",
    truncation=True,
    max_length=128
)

outputs = model.generate(
    **inputs,
    max_length=128,
    num_beams=4,
    early_stopping=True,
    pad_token_id=tokenizer.pad_token_id,
    eos_token_id=tokenizer.eos_token_id,
    decoder_start_token_id=tokenizer.pad_token_id
)

answer = tokenizer.decode(
    outputs[0],
    skip_special_tokens=True
)

print(answer)

The original local inference script used beam search with:

num_beams = 4
max_length = 128
early_stopping = True

Technical Specifications

Model Architecture

T5's encoder and decoder share the token embedding matrix.

Tokenizer

Checkpoint Size

The final Safetensors checkpoint is approximately 122 MB and contains 30.55 million FP32 parameters.

The archived checkpoint contains 89 stored tensors. Tied T5 embeddings are represented through the shared embedding weights.

Reproducibility Notes

The original project archive contains the main components needed to reconstruct the training pipeline:

• —Raw question/answer dataset.
• —Processed Hugging Face Dataset representation.
• —Tokenizer-training code.
• —Saved tokenizer.
• —Model configuration.
• —Model-training code.
• —Local inference code.
• —Final model checkpoint.
• —Generation configuration.
• —Saved training arguments.

However, the archive does not contain sufficient information to reconstruct the exact original compute environment, hardware configuration, wall-clock training time, energy consumption, or carbon emissions. These values are therefore intentionally not estimated in this model card.

Historical Context

This model was developed in October 2024 as an early experiment in building a very small Spanish conversational Transformer for a constrained educational use case.

The project is particularly useful as an example of training a compact T5-style model from scratch and also illustrates several practical issues that arise when building language models with very small custom datasets, including vocabulary sizing, data consistency, evaluation leakage, token configuration, and the distinction between model-generated information and data retrieved from an authoritative backend.

Model Card Authors

Lenin Villanueva

Model Card Contact

Hugging Face: leninangelov