telecomadm1145/Kiseki-1.1-0.3B

Model Card for Kiseki-1.1-0.3B

An updated iteration of Kiseki-0.3B with a revised dataset mix, yielding substantial improvements across EN↔ZH and JA↔EN directions.

Note: Kiseki-1.1 is not fine-tuned from Kiseki-0.3B; it is trained independently with a different dataset configuration.

Model Details

Model Description

• —Developed by: telecomadm1145
• —Model type: Transformer-Mamba2 Seq2Seq
• —Language(s) (NLP): Japanese (ja), Chinese (zh), English (en)
• —License: MIT
• —Parameters: 0.3B

⚠️ Important: UL2 Paradigm Token Required

Due to an oversight during seq2seq fine-tuning, the encoder was trained with a UL2 paradigm sentinel token prepended to every input. You must prepend token ID `65662` to the encoder `input_ids` at inference time. Omitting it will produce significantly degraded output.

ul2_token_id = 65662
input_ids = tokenizer.encode("<|2zh|>" + text + "<eos>")
input_ids = [ul2_token_id] + input_ids  # required prepend

How to Get Started

Prompt format: <|2zh|>/<|2ja|>/<|2en|>{source text}<eos>

import torch
from transformers import AutoModelForSeq2SeqLM, PreTrainedTokenizerFast

repo_id = "telecomadm1145/Kiseki-1.1-0.3B"
device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = PreTrainedTokenizerFast.from_pretrained(repo_id)
model = AutoModelForSeq2SeqLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
    dtype=torch.float32
)
model.to(device)

text = "おはようございます。"

ul2_token_id = 65662
input_ids = [ul2_token_id] + tokenizer.encode("<|2zh|>" + text + "<eos>")
input_tensor = torch.tensor([input_ids]).to(device)

output_ids = model.generate(input_tensor, max_new_tokens=256)
result = tokenizer.decode(output_ids[0], skip_special_tokens=True)
print(f"Translation: {result}")

Training Data

Metrics

Table 1: Multi-Model Comparison (ja → zh)

[Note 1] Methodology Variance in SHARD-00134 COMET Evaluation: Evaluated on SHARD-00134 at the sentence level (the 200 raw document pairs were split into 3,197 individual sentences). Sentence-level evaluation reduces length penalties and aligns semantic embeddings more tightly, resulting in a higher absolute score than document-level evaluation.

Table 2: Kiseki-0.3B vs. Kiseki-1.1-0.3B — All Directions

Values shown as 0.3B / 1.1-0.3B; bold marks the better score per cell.