mrshu/tildeopen-30b-64k-sk-alpaca-slovak-it

TildeOpen-30B-64K Alpaca Slovak IT

TildeOpen-30B-64K Alpaca Slovak IT is an instruction-tuned Slovak assistant model based on TildeAI/TildeOpen-30b-64k. It was trained with a LoRA supervised fine-tuning recipe on Slovak Alpaca-style instruction data and then merged into full model weights for direct transformers loading.

This repository contains the merged full-weight checkpoint. The adapter-only artifact is available separately at `mrshu/tildeopen-30b-64k-sk-alpaca-slovak-it-lora`.

The model is designed to answer in Slovak by default while still attempting to follow explicit user requests for another language or a specific output format.

Intended Use

This model is intended for Slovak instruction following, Slovak question answering, drafting, rewriting, summarization-style prompts, and general assistant workflows where Slovak is the primary language.

It can also respond to English prompts and translation-style requests, but language control is not guaranteed. Use additional application-level checks for strict formatting, policy compliance, or high-reliability translation.

Do not use this model as the sole source for medical, legal, financial, safety, or other high-stakes decisions. It has not been safety aligned, red-teamed, or moderated for production deployment.

Model Details

Training Data

The model was instruction-tuned on `saillab/alpaca-slovak-cleaned`, a Slovak Alpaca-style instruction dataset.

Dataset preparation converted each example into a chat-style conversation with system, user, and assistant messages. Empty instruction/output examples were excluded, and duplicate instruction/input/output triples were removed across the prepared splits.

This run trained from the prepared train split and used Axolotl val_set_size: 0.02 for evaluation. The data preparation used dataset revision 058172466eb1d6a28b161f29c74350911d154161.

Each training conversation used this system prompt:

Si užitočný asistent. Riaď sa jazykom a požadovaným formátom používateľa. Ak používateľ nežiada iný jazyk, odpovedaj po slovensky.

Prompt Format

The training run used a simple Slovak chat format:

<|endoftext|>Systém: {system}

Používateľ: {user}
Asistent: {assistant}<|endoftext|>

For generation, append Asistent: after the user message.

Training Recipe

The model was trained with supervised fine-tuning using PEFT LoRA. Only the assistant turns were included in the training loss.

Results

Final training loss was 0.9799.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "mrshu/tildeopen-30b-64k-sk-alpaca-slovak-it"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

system = (
    "Si užitočný asistent. Riaď sa jazykom a požadovaným formátom "
    "používateľa. Ak používateľ nežiada iný jazyk, odpovedaj po slovensky."
)
user = "Stručne vysvetli, čo je LoRA."

prompt = f"<|endoftext|>Systém: {system}\n\nPoužívateľ: {user}\nAsistent:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    do_sample=True,
    temperature=0.7,
    top_p=0.95,
)

generated = outputs[0][inputs["input_ids"].shape[-1]:]
print(tokenizer.decode(generated, skip_special_tokens=True))

Limitations

• —The supervised fine-tuning data is translated instruction data, so the model may inherit translation artifacts, unnatural phrasing, or source-dataset biases.
• —The checkpoint is a merged LoRA fine-tune. Runtime behavior may differ slightly from applying the adapter dynamically, depending on the loading stack and inference backend.
• —The model is biased toward Slovak responses. Explicit requests for another language may work, but they are not guaranteed.
• —Strict JSON, exact labels, citations, and other constrained formats should be validated outside the model.
• —The model may hallucinate facts, produce unsafe content, or follow malicious instructions.

Training Framework

• —Axolotl 0.16.1
• —PEFT 0.19.1
• —Transformers 5.5.0
• —PyTorch 2.11.0+cu126
• —Datasets 4.5.0
• —Tokenizers 0.22.2