tiny-random/gpt-oss-bf16

1---2library_name: transformers3pipeline_tag: text-generation4inference: true5widget:6  - text: Hello!7    example_title: Hello world8    group: Python9base_model:10- openai/gpt-oss-120b11---12 13This tiny model is for debugging. It is randomly initialized with the config adapted from [openai/gpt-oss-120b](https://huggingface.co/openai/gpt-oss-120b).14 15Note: This model is in BF16; quantized MXFP4 FFN is not used.16 17### Example usage:18 19- vLLM20 21```bash22vllm serve tiny-random/gpt-oss-bf1623```24 25- Transformers26 27```python28import torch29from transformers import pipeline30 31model_id = "tiny-random/gpt-oss-bf16"32 33pipe = pipeline(34    "text-generation",35    model=model_id,36    torch_dtype=torch.bfloat16,37    device_map="cuda"38)39 40messages = [41    {"role": "user", "content": "Explain quantum mechanics clearly and concisely."},42]43 44outputs = pipe(45    messages,46    max_new_tokens=16,47)48print(outputs[0]["generated_text"][-1])49```50 51### Codes to create this repo:52 53```python54import json55 56import torch57from huggingface_hub import hf_hub_download58from transformers import (59    AutoConfig,60    AutoModelForCausalLM,61    AutoProcessor,62    AutoTokenizer,63    GenerationConfig,64    GptOssForCausalLM,65    pipeline,66    set_seed,67)68 69source_model_id = "openai/gpt-oss-120b"70save_folder = "/tmp/tiny-random/gpt-oss-bf16"71 72processor = AutoProcessor.from_pretrained(source_model_id)73processor.save_pretrained(save_folder)74 75with open(hf_hub_download(source_model_id, filename='config.json', repo_type='model'), 'r') as f:76    config_json = json.load(f)77config_json.update({78    "head_dim": 32,79    "hidden_size": 32,  # required by Mxfp4GptOssExperts codes80    "intermediate_size": 64,81    "layer_types": ["sliding_attention", "full_attention"],82    "num_attention_heads": 2,83    "num_hidden_layers": 2,84    "num_key_value_heads": 1,85    "num_local_experts": 32,86    "tie_word_embeddings": True,87})88quantization_config = config_json['quantization_config']89del config_json['quantization_config']90with open(f"{save_folder}/config.json", "w", encoding='utf-8') as f:91    json.dump(config_json, f, indent=2)92 93config = AutoConfig.from_pretrained(save_folder)94print(config)95torch.set_default_dtype(torch.bfloat16)96model = AutoModelForCausalLM.from_config(config)97torch.set_default_dtype(torch.float32)98model.generation_config = GenerationConfig.from_pretrained(99    source_model_id, trust_remote_code=True,100)101set_seed(42)102with torch.no_grad():103    for name, p in sorted(model.named_parameters()):104        torch.nn.init.normal_(p, 0, 0.1)105        print(name, p.shape)106model.save_pretrained(save_folder)107 108# mxfp4109from transformers.quantizers.quantizer_mxfp4 import Mxfp4HfQuantizer110# model = AutoModelForCausalLM.from_pretrained(save_folder, trust_remote_code=True, torch_dtype=torch.bfloat16, quantization_config=quantization_config)111# model.save_pretrained(save_folder, safe_serialization=True)112```