num1notsvn/wicara-56m-base

Wicara 56M Base

Weight-efficient Indonesian Conversational Architecture, Research Artifact

A 56-million-parameter Indonesian language model that we trained from scratch on a single consumer laptop (NVIDIA RTX 4050 6 GB) over 1.12 billion tokens of curated text.

Wicara (from Sanskrit vicāra): speech, discourse.

Data, tokenizer, architecture, and training pipeline were built directly in PyTorch — no off-the-shelf model, no external API. This is the foundational base model before SFT. For the conversational version, see num1notsvn/wicara-56m-chat.

• —Author: Bagus Ardin Prayoga (@num1notsvn)
• —Language: Indonesian (id)
• —Parameters: 56.0M (45.5M non-embedding)
• —Architecture: LLaMA-style (Pre-norm RMSNorm, RoPE, SwiGLU, Grouped-Query Attention, Tied Embeddings)
• —License: Apache-2.0
• —Source Code: GitHub - bagusardin25/WicaraLLM

Architecture Specifications

Pretraining Details

Dataset Composition

Trained on 1.12 billion tokens (3,985,535 documents) curated exclusively from natural Indonesian texts:

Training Progression & Hardware

• —Hardware: Single laptop with NVIDIA GeForce RTX 4050 Laptop GPU (6 GB VRAM).
• —Duration: 11.1 hours (17,010 steps, 1 epoch, 100% corpus coverage).
• —Throughput: 28,017 tokens/second.
• —Precision: bf16 mixed precision with AdamW ($\beta1=0.9, \beta2=0.95$, cosine decay with warmup).
• —Validation Loss: 3.0505 (Perplexity: 21.2).
• —Training Loss: 2.937 (generalization gap: 0.12).

Quickstart & Usage

[!NOTE] As a base model, wicara-56m-base is trained for next-token prediction (text completion), not multi-turn instruction following. If you are looking for an interactive conversational assistant, please use num1notsvn/wicara-56m-chat.

Text Completion Example

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "num1notsvn/wicara-56m-base"
device = "cuda" if torch.cuda.is_available() else "cpu"

# Load tokenizer and base model
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16 if torch.cuda.is_available() else torch.float32,
).to(device)

# Provide a prefix prompt for completion
prompt = "Indonesia adalah sebuah negara kepulauan yang"
inputs = tokenizer(prompt, return_tensors="pt").to(device)

# Generate continuation
outputs = model.generate(
    **inputs,
    max_new_tokens=64,
    temperature=0.7,
    top_p=0.9,
    repetition_penalty=1.15,
    do_sample=True,
    eos_token_id=tokenizer.eos_token_id,
)

continuation = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(continuation)

Intended Use & Fine-Tuning

This base model is intended as a foundation for:

• —Downstream Fine-Tuning: Supervised fine-tuning (SFT) for Indonesian NLP tasks (sentiment analysis, intent classification, named entity recognition, or custom domain adaptation).
• —Edge SLM Research: Exploring the boundaries of lightweight language models (<100M parameters) under strict compute and VRAM budgets.
• —Academic & Educational Use: Studying transformer pretraining dynamics from scratch.

Limitations

• —No Instruction Tuning: This model does not understand conversational turns or system prompts out of the box.
• —Hallucination & Factual Consistency: Small-capacity models (56M parameters) prioritize surface-level linguistic fluency and cannot reliably serve as factual knowledge repositories without external retrieval (RAG).
• —Context Length: The native maximum context length is 512 tokens.

Citation & License

Released under the Apache-2.0 License.

@misc{ardin2026wicara,
  title={Wicara: A Weight-efficient Indonesian Conversational Architecture Built from Scratch},
  author={Bagus Ardin Prayoga},
  year={2026},
  publisher={Hugging Face},
  howpublished={\url{https://huggingface.co/num1notsvn/wicara-56m-base}}
}