SwayAlgo/SwayAlgo-Finance-gemma-4-E4B-it-1

SwayAlgo Finance Gemma 4 E4B-it 1

SwayAlgo Finance Gemma 4 E4B-it 1 is a finance-oriented language model adaptation built by SwayAlgo on top of google/gemma-4-E4B-it.

This repository contains the merged model weights for the verified SwayAlgo Finance Gemma 4 E4B-it 1 run that reached 45.45% accuracy on our full local BhashaBench-Finance evaluation file.

The package includes:

• —Full merged model weights
• —Tokenizer and chat-template files
• —Evaluation summary
• —Per-question predictions
• —Mistake file

This model is not BharatGen FinanceParam. FinanceParam is a separate model from BharatGen. This repository documents SwayAlgo's own merged Gemma 4 E4B-it finance model and compares it transparently against the FinanceParam benchmark numbers shown on BharatGen's public model card.

The comparison should be read with model size in mind. FinanceParam is a smaller model based on Param-1-2.9B-Instruct, while SwayAlgo Finance Gemma 4 E4B-it 1 uses the larger Gemma 4 E4B-it base. The benchmark tables report observed accuracy under the stated evaluation setup; they are not intended to claim equal model size, equal compute cost, or equal deployment efficiency.

Motivation

Indian finance assistants need more than generic language fluency. They need practical reasoning across quantitative aptitude, accounting, banking, commerce, finance education, policy, regulatory, and business-operation style questions.

This run was designed around one focused objective: improve BhashaBench-Finance-style multiple-choice performance while keeping the training process inspectable and reproducible.

The training strategy was:

1. 1.Start with google/gemma-4-E4B-it.
2. 2.Fine-tune a compact LoRA adapter instead of directly updating every base-model weight.
3. 3.Merge the trained LoRA adapter back into the Gemma 4 E4B-it base model.
4. 4.Analyze previous mistakes.
5. 5.Build a targeted anti-bias training corpus.
6. 6.Evaluate the merged model on the full BhashaBench-Finance file and save all predictions.

Model Architecture

SwayAlgo Finance Gemma 4 E4B-it 1 inherits its base architecture from google/gemma-4-E4B-it.

Text model details from the local base model configuration:

• —Base model: google/gemma-4-E4B-it
• —Architecture: Gemma4ForConditionalGeneration
• —Model type: gemma4
• —Text hidden size: 2560
• —Text intermediate size: 10240
• —Text hidden layers: 42
• —Attention heads: 8
• —Key-value heads: 2
• —Max position embeddings: 131072
• —Sliding window: 512
• —Vocabulary size: 262144
• —Precision: bfloat16
• —Training adaptation method: LoRA
• —LoRA rank used during training: 4
• —LoRA alpha used during training: 8
• —LoRA dropout: 0
• —PEFT type during training: LoRA
• —Task type: causal language modeling
• —Published artifact: full merged model

Training was done with LoRA, then the learned LoRA update was merged into the Gemma 4 E4B-it base weights. The uploaded model is intended to load directly as a standard causal language model.

Repository Layout

The Hugging Face repository is organized as a merged-model release with model files at the root and evaluation artifacts in separate folders:

.
|-- base-model-evaluations/
|-- evaluations/
|-- .gitattributes
|-- README.md
|-- chat_template.jinja
|-- config.json
|-- generation_config.json
|-- model-00001-of-00004.safetensors
|-- model-00002-of-00004.safetensors
|-- model-00003-of-00004.safetensors
|-- model-00004-of-00004.safetensors
|-- model.safetensors.index.json
|-- tokenizer.json
|-- tokenizer_config.json
`-- tokenizer_config.json.bak

The model is uploaded as a merged model, so users can load it directly with Transformers without separately attaching a LoRA adapter.

Data Preparation

The training data was prepared as a focused finance-reasoning corpus aligned with the shape of the evaluation task. The goal was to improve option selection, numerical reasoning, and consistency across common Indian finance benchmark patterns without turning the model into a memorization artifact.

The preparation flow combined structured finance examples, controlled multiple-choice transformations, and targeted correction passes from earlier evaluation traces. Questions were normalized into a compact instruction format, answer choices were balanced across labels, and exact evaluation-input overlap was filtered out before training.

At a high level, the process emphasized:

• —Finance, accounting, aptitude, and business-reasoning coverage.
• —Multiple-choice consistency with single-letter answer supervision.
• —Balanced answer labels to reduce option bias.
• —Targeted reinforcement on previously weak patterns.
• —Reproducible JSONL generation with saved evaluation artifacts.

Approximate training exposure for this run:

• —Optimizer steps: 8509
• —Per-device batch size: 1
• —Gradient accumulation: 8
• —Approximate examples processed: 68072
• —Epochs: 1
• —Max sequence length: 1024

Training Setup

• —Base model: google/gemma-4-E4B-it
• —Training framework: Unsloth + TRL SFTTrainer + PyTorch
• —Adaptation framework used during training: PEFT LoRA
• —Precision: bfloat16
• —Epochs: 1
• —Learning rate: 3e-5
• —LoRA rank: 4
• —LoRA alpha: 8
• —Gradient accumulation steps: 8
• —Save steps: 100
• —Final checkpoint: checkpoint-8509
• —Final epoch: 1.0
• —Hardware used: NVIDIA GeForce RTX 5060 Ti 16 GB

Training environment variables used for the LoRA stage before merge:

UNSLOTH_BASE_MODEL=/home/swayalgo-pc-01/swayalgo-models/base/gemma-4-E4B-it
UNSLOTH_OUTPUT_DIR=/home/swayalgo-pc-01/swayalgo-models/swayalgo-finance-gemma-4-e4b-it-1-adapter
UNSLOTH_MAX_SEQ_LENGTH=1024
UNSLOTH_LORA_RANK=4
UNSLOTH_LORA_ALPHA=8
UNSLOTH_GRAD_ACCUM=8
UNSLOTH_EPOCHS=1
UNSLOTH_LEARNING_RATE=3e-5
UNSLOTH_SAVE_STEPS=100
UNSLOTH_LOGGING_STEPS=1
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
TOKENIZERS_PARALLELISM=false

Inference Example

Because this repository is a merged model, it can be loaded directly with Transformers. PeftModel is not required for normal inference.

Gemma 4 support is expected in recent Transformers builds. If your environment does not recognize model_type: gemma4, upgrade the inference stack before loading:

pip install -U "transformers>=5.13.1" accelerate safetensors torch

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "Swayalgo/SwayAlgo-Finance-gemma-4-E4B-it-1"

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16 if torch.cuda.is_available() else torch.float32,
    device_map="auto",
    trust_remote_code=True,
)

messages = [
    {
        "role": "user",
        "content": "A company has assets of Rs. 500000 and liabilities of Rs. 180000. What is owner's equity?",
    }
]

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    return_tensors="pt",
).to(model.device)

with torch.no_grad():
    output = model.generate(
        inputs,
        max_new_tokens=128,
        temperature=0.2,
        do_sample=False,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Evaluation Method

Evaluation was performed on the merged model using an internal multiple-choice scoring harness. The harness prompts the model on BhashaBench-Finance rows and extracts the final answer letter A/B/C/D/E.

Evaluated on the public BhashaBench-Finance benchmark dataset. The local Gemma 4 E4B-it base and SwayAlgo merged-model evaluations used runtime 4-bit loading through Unsloth/bitsandbytes on the same local evaluation setup. Published comparison values for smaller reference models and FinanceParam are reproduced from the BharatGen FinanceParam model card.

Evaluation artifacts in this repository:

• —evaluations/summary.json
• —evaluations/predictions.jsonl
• —evaluations/mistakes.jsonl

Benchmarks

Overall BhashaBench-Finance Performance

The reference-model and FinanceParam values below are reproduced from the public BharatGen FinanceParam model card. Gemma 4 E4B-it base values are computed from gemma-4-e4b-it-base-full. SwayAlgo merged-model values are from the included local evaluation artifacts: evaluations/summary.json, evaluations/predictions.jsonl, and evaluations/mistakes.jsonl. The local Gemma 4 E4B-it base and SwayAlgo rows were both evaluated with runtime 4-bit loading, so those two rows are internally comparable under the same local runtime settings.

Evaluation size:

Expected answer distribution:

Predicted answer distribution:

Domain-Wise Performance

Eval Questions is the number of local BhashaBench-Finance rows used to compute the Gemma 4 E4B-it base and SwayAlgo scores for that domain.

Difficulty-Level Performance

Eval Questions is the number of local BhashaBench-Finance rows used to compute the Gemma 4 E4B-it base and SwayAlgo scores for that difficulty level.

Question-Type Performance

Eval Questions is the number of local BhashaBench-Finance rows used to compute the Gemma 4 E4B-it base and SwayAlgo scores for that question type. The single Essay row in the local artifact is omitted from this table because the comparison model card does not report an essay row.

Benchmark interpretation note: SwayAlgo numbers are truthful to the local merged-model evaluation artifacts included with this repository. Gemma 4 E4B-it base and SwayAlgo numbers are computed with local evaluation artifacts using runtime 4-bit loading. The other reference-model and FinanceParam numbers are reproduced from the published FinanceParam model card. Treat this as a reported-results comparison unless all models are rerun through the exact same benchmark checkout, prompt, decoding parameters, runtime precision or quantization mode, and answer-extraction script.

Limitations

• —This model is not a certified tax, legal, accounting, or investment advisor.
• —It can make mistakes on hard reasoning, rearrangement, reading comprehension, and mathematics-heavy questions.
• —The training process targeted benchmark-style multiple-choice behavior, so open-ended finance quality should be evaluated separately.
• —Benchmark comparison depends on prompt format, decoding settings, and answer extraction logic.
• —The base model license and Apache 2.0 terms must be followed.

References

• —Base model: google/gemma-4-E4B-it
• —Benchmark dataset: bharatgenai/BhashaBench-Finance
• —Comparison model: bharatgenai/FinanceParam
• —Training framework: Unsloth
• —Adapter framework used during training: PEFT
• —SFT framework: TRL

Citation

If you use this model or its reported evaluation artifacts, cite it as:

@misc{swayalgo_finance_gemma_4_e4b_it_1_2026,
  title        = {SwayAlgo Finance Gemma 4 E4B-it 1},
  author       = {SwayAlgo},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/Swayalgo/SwayAlgo-Finance-gemma-4-E4B-it-1}},
  note         = {Merged finance-domain model based on google/gemma-4-E4B-it}
}