tiny-random/gemma-3

1---2library_name: transformers3pipeline_tag: image-text-to-text4inference: true5widget:6  - text: Hello!7    example_title: Hello world8    group: Python9---10 11This tiny model is for debugging. It is randomly initialized with the config adapted from [google/gemma-3-27b-it](https://huggingface.co/google/gemma-3-27b-it).12 13### Example usage:14 15```python16from transformers import pipeline17model_id = "tiny-random/gemma-3"18pipe = pipeline(19    "image-text-to-text", model=model_id, device="cuda",20    trust_remote_code=True, max_new_tokens=3,21)22messages = [23    {24        "role": "system",25        "content": [{"type": "text", "text": "You are a helpful assistant."}]26    },27    {28        "role": "user",29        "content": [30            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},31            {"type": "text", "text": "What animal is on the candy?"}32        ]33    }34]35output = pipe(text=messages, max_new_tokens=5)36print(output)37```38 39### Codes to create this repo:40 41```python42import torch43 44from transformers import (45    AutoConfig,46    AutoModelForCausalLM,47    AutoProcessor,48    AutoTokenizer,49    Gemma3ForConditionalGeneration,50    GenerationConfig,51    pipeline,52    set_seed,53)54 55source_model_id = "google/gemma-3-27b-it"56save_folder = "/tmp/tiny-random/gemma-3"57 58processor = AutoProcessor.from_pretrained(59    source_model_id, trust_remote_code=True,60)61processor.save_pretrained(save_folder)62 63config = AutoConfig.from_pretrained(64    source_model_id, trust_remote_code=True,65)66config.text_config.hidden_size = 3267config.text_config.intermediate_size = 12868config.text_config.head_dim = 3269config.text_config.num_attention_heads = 170config.text_config.num_key_value_heads = 171config.text_config.num_hidden_layers = 272config.text_config.sliding_window_pattern = 273config.vision_config.hidden_size = 3274config.vision_config.num_hidden_layers = 275config.vision_config.num_attention_heads = 176config.vision_config.intermediate_size = 12877model = Gemma3ForConditionalGeneration(78    config,79).to(torch.bfloat16)80for layer in model.language_model.model.layers:81    print(layer.is_sliding)82model.generation_config = GenerationConfig.from_pretrained(83    source_model_id, trust_remote_code=True,84)85set_seed(42)86with torch.no_grad():87    for name, p in sorted(model.named_parameters()):88        torch.nn.init.normal_(p, 0, 0.5)89        print(name, p.shape)90model.save_pretrained(save_folder)91```