alakxender/gemma-3-270m-dhivehi-pt

Gemma-3-270M Dhivehi

Compact Dhivehi (ދިވެހި) pretrained model based on google/gemma-3-270m, trained on a large corpus of Dhivehi text data including news articles, Wikipedia content, and general web text.

Note: This model is specifically pretrained on Dhivehi text and provides a strong foundation for further fine-tuning on specific tasks or direct use for text generation.

Model details

• —Base: google/gemma-3-270m
• —Language: Dhivehi
• —Training data: Large corpus of Dhivehi text including:
• —News articles (Dhivehi news corpus)
• —Wikipedia articles
• —General web content (FineWeb-2)
• —Glot500 Dhivehi dataset
• —Training method: Pretraining with causal language modeling
• —Training data size: Large corpus combining multiple Dhivehi datasets
• —Supported tasks:
• —Text generation in Dhivehi
• —Foundation for fine-tuning on specific tasks

Training Data

The model was pretrained on a comprehensive corpus of Dhivehi text data:

• —Random Articles: News articles from diffrent sources
• —Dhivehi News Corpus: General news content in Dhivehi
• —Wikipedia (dv): Dhivehi Wikipedia articles
• —FineWeb-2: Filtered web content in Dhivehi
• —Glot500 (div-thaa): Multilingual dataset with Dhivehi content

Intended use

• —As a foundation model for fine-tuning on specific Dhivehi tasks
• —For general text generation in Dhivehi
• —For research and development of Dhivehi language models

How to use

from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
import torch

# Load model
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
import torch

# Load model
model_path = "alakxender/gemma-3-270m-dhivehi-pt"
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    torch_dtype="auto",
    device_map="auto",
)

# Method 1: Direct text generation
prompt = "ދިވެހިރާއްޖެއަކީ"

# Tokenize input
inputs = tokenizer(
    prompt, 
    return_tensors="pt", 
    padding=True
)

# Move inputs to the same device as the model
if torch.cuda.is_available():
    inputs = {k: v.to(model.device) for k, v in inputs.items()}

# Generate content
with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=150,
        # num_return_sequences=1,
        # temperature=0.8,
        # do_sample=True,
        # pad_token_id=tokenizer.eos_token_id,
        # eos_token_id=tokenizer.eos_token_id,
        # repetition_penalty=1.1,
    )

# Decode generated text
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)

# Extract only the newly generated part
generated_only = generated_text[len(prompt):].strip()

print(f"Generated: {generated_only}")

# `gemma-3-270m-dhivehi-pt`: އަޅުގަނޑުމެންގެ ގޮނޑުދޮށްތަކާއި ފަރުބަދަތަކާއި އުތުރު ހިންދުސްތާނުގެ ވަކިވަކި ހިސާބުގައި ދިރިއުޅޭ ދިވެހިންގެ މެދުގައި އޮންނަ ގުދުރަތީ ކަންކަން ދެކިލުމުގެ ފުރުޞަތު އޮތް ޤައުމެއްކަމުގައި ދުވަހަކުވެސް ހިޔެއްނުކުރާނެއެވެ. 33 އަހަރުގެ ވެރިކަމުން މިދެންނެވި
# `google/gemma-3-270m`:  ވެސް އޭނާގެ ރާއްޖެއެއްކަމަށް ވެރިކަން ނުވަތަ ޓީމް ނޭޝަނަލްގެ ބައިވެރިވަރމް އޭޝިޔާގެ ހައިސިއްޔަތް ބޯޑިޔަށް ނުހުންދާ ކަމަށް ލިޔުއްވައިގައެވެ. ދިވެހިންނާއި މިއީ އެއާ އޭޝިޔާގެ ބަޔާންކޮށް ނިޒާމް ނެތުމުގައި ދާއިރާއަށް އޭޝިޔާއަށް ދެއްވައިދޭ މަލިވާރުކަމަށް

Generation Parameters

• —`max_new_tokens`: Controls the length of generated text (64-512 recommended)
• —`temperature`: Controls randomness (0.1-1.0, higher = more creative)
• —`top_p`: Nucleus sampling parameter (0.1-1.0)
• —`top_k`: Top-k sampling parameter (1-100)
• —`do_sample`: Boolean flag to enable/disable sampling

Limitations

• —Generated content may not always be factually accurate
• —Quality depends on the clarity and specificity of input prompts
• —Context window limitations for very long inputs
• —The model is specifically pretrained on Dhivehi text and may require fine-tuning for specific tasks
• —No instruction-following capabilities (this is a pretrained model, not instruction-tuned)