nischay185/konkani-qwen2-1.5b-v3-full

Konkani Qwen2-1.5B V3 Full

Konkani Qwen2-1.5B V3 Full is the standalone full-weight version of nischay185/konkani-qwen2-1.5b-v3.

It contains the same V3 model weights as the original V3 LoRA/PEFT release, but the adapter has been merged into the underlying Konkani SFT model and released as a standalone model.

Unlike the V3 adapter release, this repository does not require the original Konkani SFT model or PEFT to be loaded separately.

The model is designed to improve useful reasoning and task-following capabilities while retaining the Konkani language behavior learned during the original SFT stage.

Model Lineage

V3 was not trained directly from the original Qwen2-1.5B base model.

The model lineage is:

Qwen/Qwen2-1.5B
        ↓
nischay185/konkani-qwen2-1.5b
Original Konkani SFT
        ↓
V3 LoRA / PEFT training
        ↓
nischay185/konkani-qwen2-1.5b-v3
V3 Adapter
        ↓
Adapter merged into base model
        ↓
nischay185/konkani-qwen2-1.5b-v3-full
Standalone V3 Full

The original V3 adapter is available as:

nischay185/konkani-qwen2-1.5b-v3

This repository is the standalone version:

nischay185/konkani-qwen2-1.5b-v3-full

What Is Different From the V3 Adapter?

The underlying model behavior and V3 training are the same.

The primary difference is the model format.

V3 Adapter

The adapter release contains only the LoRA adapter weights and requires:

nischay185/konkani-qwen2-1.5b
+
nischay185/konkani-qwen2-1.5b-v3

PEFT is required to attach the adapter to the original Konkani SFT model.

V3 Full

The full release contains the resulting standalone model weights:

nischay185/konkani-qwen2-1.5b-v3-full

It can be loaded directly with Transformers without separately loading the original Konkani SFT model or the V3 adapter.

The V3 Full model is therefore intended to provide a simpler deployment and inference experience.

Why V3?

The main objective of V3 is to improve the reasoning capability of a low-resource Konkani language model without replacing the language-specific knowledge and behavior learned during the original Konkani SFT stage.

The training approach was inspired by English-Pivoted Chain-of-Thought (CoT) reasoning for extremely low-resource languages.

The basic idea is:

Konkani question
       ↓
English reasoning trace
       ↓
Konkani final answer

English is used as an intermediate reasoning language because the underlying model has stronger English reasoning representations than Konkani.

The final response is still expected to be in Konkani.

Training Data

V3 was trained using a mixed dataset containing both original Konkani SFT examples and reasoning-oriented CoT examples.

The final dataset contains approximately:

• —50% original SFT replay
• —50% reasoning/CoT data

The SFT replay examples were included to help preserve the original model's Konkani behavior while introducing the new reasoning-oriented training examples.

Training Configuration

V3 was trained using LoRA/PEFT on top of the original Konkani SFT model.

LoRA rank (r):              16
LoRA alpha:                 32
LoRA dropout:               0.05

Target modules:
- q_proj
- k_proj
- v_proj
- o_proj
- gate_proj
- up_proj
- down_proj

Learning rate:              2e-4
Scheduler:                  Cosine
Warmup ratio:               0.05
Weight decay:               0.01

Micro batch size:           2
Gradient accumulation:      8
Effective batch size:       16

Maximum sequence length:    512
Training epochs:            1
Optimizer:                  paged_adamw_8bit
Seed:                       42

After training, the V3 adapter was merged with the original Konkani SFT model to produce this standalone release.

Training Results

The V3 training run completed successfully with:

Global steps:       750
Training loss:      0.5795
Training runtime:   ~0.83 hours

Validation loss was recorded during training:

Step 200:   0.592825
Step 400:   0.556194
Step 600:   0.537714

Evaluation

V3 was evaluated side-by-side against the original Konkani SFT model.

The comparison used:

Original SFT:
nischay185/konkani-qwen2-1.5b

V3:
nischay185/konkani-qwen2-1.5b
+
nischay185/konkani-qwen2-1.5b-v3

The V3 Full release contains the resulting standalone weights from the latter configuration.

Both models were given the same questions, system instruction, generation settings, and random seed.

The evaluation contained 39 questions across 13 categories.

Retention Categories

These categories were used to check whether V3 retained the behavior of the original Konkani SFT model:

• —Identity
• —General Knowledge
• —History
• —Culture & Festivals
• —Food
• —Translation
• —Conversation
• —Polite Refusal

Reasoning Categories

These categories were used to examine reasoning and task-following:

• —Arithmetic
• —Temporal Reasoning
• —Multi-step Reasoning
• —Reading Comprehension
• —Coreference

The evaluation was manually judged rather than using simple string matching.

Answers were considered based on factors such as:

• —factual correctness
• —relevance to the question
• —natural Konkani
• —following the requested task
• —arithmetic correctness
• —temporal reasoning
• —reading comprehension
• —coreference resolution
• —unnecessary hallucination
• —retention of ordinary Konkani behavior

The purpose of the evaluation is not only to ask whether V3 "reasons better", but whether it can improve useful reasoning while retaining the Konkani capabilities of the original SFT model.

How to Load V3 Full

Unlike the adapter release, the V3 Full model can be loaded directly.

Installation

pip install -U transformers accelerate torch

Inference

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL_ID = "nischay185/konkani-qwen2-1.5b-v3-full"

tokenizer = AutoTokenizer.from_pretrained(
    MODEL_ID,
    trust_remote_code=True,
)

if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

dtype = torch.float16 if torch.cuda.is_available() else torch.float32

model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    dtype=dtype,
    device_map="auto",
    trust_remote_code=True,
)

model.eval()

No separate PeftModel loading is required.

No separate download of nischay185/konkani-qwen2-1.5b is required for inference.

Example Prompt Format

The evaluation used the following system instruction:

तुमी एक कोंकणी भाशेचो सहाय्यक,फक्त कोंकणी भाशेंत जाप दिवची,मराठी वा हिंदी वापरूं नाका.

The model was prompted using the Qwen chat format:

<|im_start|>system
[system instruction]
<|im_end|>
<|im_start|>user
[user question]
<|im_end|>
<|im_start|>assistant

For reasoning-oriented examples, V3 may produce an English reasoning trace followed by a Konkani answer.

A typical intended structure is:

<think>
English reasoning
</think>
Konkani final answer

The presence of a <think> section alone should not be interpreted as proof that the reasoning is correct. The final answer and task outcome should be evaluated for correctness.

Generation Settings Used During Evaluation

Seed:                  42
Max new tokens:        220
Temperature:           0.7
Top-p:                 0.9
Repetition penalty:    1.1
Sampling:              enabled

These settings are provided for reproducibility of the reported evaluation and are not necessarily optimal for every application.

Intended Use

This model is intended for:

• —research on low-resource language modeling
• —Konkani NLP experimentation
• —reasoning transfer experiments
• —evaluation of Konkani language capabilities
• —academic and educational experimentation
• —local inference and deployment of a standalone Konkani model

Limitations

This is a small 1.5B-parameter language model specialized for low-resource Konkani experimentation.

The model may:

• —produce factual errors or hallucinations
• —produce incorrect reasoning traces
• —have limited world knowledge
• —struggle with complex multi-step reasoning
• —produce mixed-language output in some situations
• —perform differently across Konkani domains and dialectal or orthographic variations

The <think> section should not be treated as a guarantee of correct reasoning.

The reported evaluation is based on a relatively small manually evaluated set of 39 questions and should not be interpreted as a comprehensive benchmark of Konkani reasoning ability.

Credits

Base architecture:

Qwen/Qwen2-1.5B

Original Konkani SFT model:

nischay185/konkani-qwen2-1.5b

V3 adapter:

nischay185/konkani-qwen2-1.5b-v3

V3 standalone model:

nischay185/konkani-qwen2-1.5b-v3-full

V3 was developed as part of a research project investigating methods for improving low-resource language model capabilities using Goan Konkani as the test case.

Related Models

Original Konkani SFT

nischay185/konkani-qwen2-1.5b

V3 LoRA Adapter

nischay185/konkani-qwen2-1.5b-v3

V3 Full Standalone

nischay185/konkani-qwen2-1.5b-v3-full

The adapter and full releases represent the same V3 model, provided in different formats for different deployment requirements.

Citation

If you use this model in research, please cite the associated work when available.

License

Please refer to the license and usage conditions of the original Qwen2-1.5B model and the original Konkani SFT model before redistributing or commercially using this model.

The V3 Full release contains merged weights derived from the original Konkani SFT model and the V3 LoRA training process. Users are responsible for complying with the applicable licenses and usage conditions of the underlying models.