justinj92/gpt-oss-nemo-20b

GPT-OSS-NEMO-20B: Multilingual Thinking Model

Model Description

GPT-OSS-NEMO-20B is a fine-tuned version of OpenAI's GPT-OSS-20B model, specifically enhanced for multilingual reasoning and thinking capabilities. This model has been trained using Supervised Fine-Tuning (SFT) on the HuggingFaceH4/Multilingual-Thinking dataset to improve its ability to reason in multiple languages while maintaining strong performance across diverse linguistic contexts.

Key Features

• —🌍 Multilingual Reasoning: Enhanced ability to think and reason in multiple languages
• —🧠 Chain-of-Thought: Improved reasoning capabilities with explicit thinking processes
• —💬 Conversational: Optimized for interactive dialogue and question-answering
• —🎯 Cross-lingual: Can reason in one language and respond in another
• —⚡ High Performance: Built on the robust 20B parameter GPT-OSS foundation

Training Details

Base Model

• —Model: openai/gpt-oss-20b
• —Parameters: 20 billion parameters
• —Architecture: GPT-OSS (Mixture of Experts)

Fine-tuning Configuration

• —Method: LoRA (Low-Rank Adaptation)
• —Rank (r): 8
• —Alpha: 16
• —Target Modules: All linear layers with specific focus on MoE expert layers
• —Target Parameters:
• —Layer 7, 15, 23 MLP experts (gateupproj, down_proj)

Training Infrastructure

• —Hardware: 4x NVIDIA H100 GPUs
• —Cloud Platform: Microsoft Azure NC-series instances
• —Training Framework: TRL (Transformers Reinforcement Learning)
• —Optimization: AdamW with cosine learning rate scheduling

Training Hyperparameters

• —Learning Rate: 2e-4
• —Batch Size: 4 per device (16 total with 4 GPUs)
• —Gradient Accumulation: 4 steps
• —Epochs: 4
• —Max Sequence Length: 2048 tokens
• —Warmup Ratio: 3%
• —LR Scheduler: Cosine with minimum LR (10% of peak)
• —Gradient Checkpointing: Enabled

Dataset

• —Name: HuggingFaceH4/Multilingual-Thinking
• —Purpose: Multilingual reasoning and thinking enhancement
• —Languages: Multiple languages including English, Spanish, Arabic, French, German, Chinese, Japanese, Korean, Hindi, Russian
• —Training Split: Full training set

Usage

Quick Start

from transformers import AutoModelForCausalLM, AutoTokenizer

# Load model and tokenizer
model = AutoModelForCausalLM.from_pretrained(
    "justinj92/gpt-oss-nemo-20b",
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("justinj92/gpt-oss-nemo-20b")

# Example: Multilingual reasoning
messages = [
    {"role": "system", "content": "reasoning language: Arabic"},
    {"role": "user", "content": "¿Cuál es la capital de Australia?"}
]

inputs = tokenizer.apply_chat_template(
    messages, 
    add_generation_prompt=True, 
    return_tensors="pt"
)

outputs = model.generate(
    inputs,
    max_new_tokens=512,
    temperature=0.6,
    do_sample=True
)

response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

Advanced Usage with Custom Reasoning Language

# Specify reasoning language in system prompt
reasoning_language = "French"  # Can be any supported language
system_prompt = f"reasoning language: {reasoning_language}"

messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": "Explain quantum computing in simple terms."}
]

Model Capabilities

Multilingual Reasoning

The model can:

• —Think and reason in a specified language (via system prompt)
• —Process questions in one language and reason in another
• —Maintain coherent logic across language boundaries
• —Provide explanations with explicit reasoning steps

Language Support

Primary languages include:

• —English (en)
• —Spanish (es)
• —Arabic (ar)
• —French (fr)
• —German (de)
• —Chinese (zh)
• —Japanese (ja)
• —Korean (ko)
• —Hindi (hi)
• —Russian (ru)

Performance

The model demonstrates improved performance in:

• —Cross-lingual reasoning tasks
• —Multi-step problem solving
• —Contextual understanding across languages
• —Maintaining coherence in multilingual conversations

Limitations

• —Performance may vary across different languages
• —Complex reasoning in low-resource languages may be limited
• —Generated content should be verified for factual accuracy
• —May exhibit biases present in the training data

Technical Specifications

• —Model Size: ~20B parameters
• —Precision: BF16 (Brain Floating Point 16-bit)
• —Memory Requirements: ~40GB VRAM for inference
• —Recommended Hardware: NVIDIA A100/H100 or similar high-memory GPUs
• —Framework Compatibility: transformers, torch, accelerate

Citation

If you use this model in your research, please cite:

@misc{gpt-oss-nemo-20b,
  title={GPT-OSS-NEMO-20B: A Multilingual Thinking Model},
  author={justinj92},
  year={2025},
  howpublished={\url{https://huggingface.co/justinj92/gpt-oss-nemo-20b}},
  note={Fine-tuned from openai/gpt-oss-20b using HuggingFaceH4/Multilingual-Thinking}
}

Acknowledgments

• —Base Model: OpenAI GPT-OSS-20B team
• —Dataset: HuggingFace H4 team for the Multilingual-Thinking dataset
• —Infrastructure: Microsoft Azure for cloud computing resources
• —Framework: Hugging Face transformers and TRL libraries

License

This model is released under the Apache 2.0 license, following the base model's licensing terms.

Model trained on August 2025 using state-of-the-art multilingual reasoning techniques.