RobbieJ/granite-3.1-8b-bfrpg-osft

Granite 3.1 8B — Basic Fantasy RPG (OSFT v2)

A domain-expert language model fine-tuned on Basic Fantasy Role-Playing Game source material using OSFT (Orthogonal Subspace Fine-Tuning), a continual learning method that preserves the base model's general knowledge while injecting new domain expertise.

This is the v2 release, trained on significantly improved synthetic data that closed 68% of the performance gap vs the base model compared to v1.

Model Details

Evaluation Results

Evaluated on 210 questions across 16 BFRPG categories, scored 0-10 by an automated LLM judge (qwen3-14b) against reference answers.

Overall Performance

Head-to-Head: OSFT v2 wins 23 / Base wins 34 / Ties 153 Win Rate (excl. ties): 40.4%

Performance Across Training Iterations

Results by Category

Key Improvement: Training Data Quality

The biggest change in v2 was not the algorithm — it was the training data:

Why OSFT?

Traditional fine-tuning (SFT/LoRA) on domain data caused significant knowledge degradation — the base model scored 5.54 vs the LoRA fine-tuned model's 4.12 (granite judge). OSFT addresses this by:

1. 1.Computing SVD of each weight matrix to identify critical dimensions
2. 2.Freezing the most critical 85% of weight dimensions
3. 3.Only updating the least critical 15% (unfreeze_rank_ratio=0.15)

This mathematically guarantees that new domain knowledge doesn't overwrite existing capabilities.

Training Details

Data

• —Source: 5 Basic Fantasy RPG PDFs (Core Rules r142, Beginner's Essentials r18, Field Guide Omnibus r4, Monster Index r7, Equipment Emporium r33)
• —Pipeline: PDF → Markdown (Docling) → Semantic chunking (466 chunks) → Q&A generation (qwen3-14b teacher) → Quality filtering → Merged dataset
• —Training examples: 5,717 instruction-response pairs (3,678 v2 high-quality + 2,039 best of v1)
• —Coverage: All 466 document chunks, 16 categories
• —Format: ChatML with system prompt, user question, and assistant answer

Hyperparameters

Training Loss

Hardware

• —GPU: NVIDIA DGX Spark (GB10 Grace Blackwell Superchip)
• —Memory: 128 GB unified CPU+GPU
• —Peak memory usage: 78.4 GB
• —Training throughput: ~75 tokens/sec
• —Total training time: ~10 hours (including ~35 min SVD initialization)

Toolchain

• —Training Hub v0.4.0 (OSFT implementation)
• —Docling (PDF to markdown conversion)
• —Custom SDG script with qwen3-14b teacher model
• —PyTorch 2.9.0+cu130

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "RobbieJ/granite-3.1-8b-bfrpg-osft"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="bfloat16", device_map="auto")

messages = [
    {"role": "system", "content": "You are an expert on Basic Fantasy Role-Playing Game (BFRPG). You provide accurate, helpful answers about BFRPG rules, character creation, combat, spells, monsters, equipment, and gameplay."},
    {"role": "user", "content": "What is the Armor Class and cost of plate mail in BFRPG?"},
]

input_ids = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)
outputs = model.generate(input_ids, max_new_tokens=512, temperature=0.7, do_sample=True)
print(tokenizer.decode(outputs[0][input_ids.shape[-1]:], skip_special_tokens=True))

Limitations

• —Remaining gap: The fine-tuned model scores -0.14 points below the base model overall. Categories like Monster Index (-2.5) and Races (-2.1) have persistent gaps.
• —Response length: Generates shorter responses than the base model, which may affect perceived completeness.
• —Numeric precision: Some specific values (prices, percentages) may be plausible but incorrect — the model learns patterns better than exact data points.
• —BFRPG-specific: Tuned for Basic Fantasy RPG only. May not generalize to other RPG systems.
• —Inherited limitations: All limitations of the base Granite 3.1 8B model apply.

Intended Use

• —BFRPG game masters looking for rules lookup assistance
• —Players needing character creation, spell, or equipment reference help
• —RPG content creators working with BFRPG material
• —Research into domain-specific fine-tuning with knowledge preservation (OSFT)
• —Educational reference for LLM fine-tuning pipelines

Ethical Considerations

This model is fine-tuned on open-source RPG content (Basic Fantasy RPG is released under the Open Game License). It generates fictional game content and should not be used for real-world decision-making.