Maincode/Maincoder-1B

<img src="https://huggingface.co/datasets/Maincode/assets/resolve/e51154e034201be1a5dad0e9c8de31d8b9f17643/maincoder_logo.png" alt="" width="1250">

**Maincoder-1B** is a code-focused language model optimized for code generation and completion tasks. The model achieves strong performance on coding benchmarks while maintaining a compact size suitable for local deployment.

Key Features

• —Code Generation: Optimized for Python code completion and generation tasks.
• —Compact Size: 1 billion parameters, lightweight enough to run on consumer hardware.
• —Deep Architecture: Modern transformer architecture with RoPE embeddings, grouped-query attention, QK normalization and high depth-to-width ratio.
• —Advanced Data Mixing: Pre-trained and mid-trained on custom data mixes developed for high-performance coding.
• —MCPO Algorithm: Fine-tuned with specialised reinforcement learning policy optimisation algorithm to improve training stability and accelerate convergence.
• —SOTA Performance: State-of-the-art performance on Python coding benchmarks HumanEval, HumanEval+ and MBPP+.

Benchmark Results

<img src="https://huggingface.co/datasets/Maincode/assets/resolve/main/performance_h.png" alt="Benchmark Performance Across Baseline LLMs" width="1050">

Model Overview

Maincoder uses a modern transformer decoder architecture with:

• —Rotary Position Embeddings: With theta of 1,000,000.
• —RMSNorm: Pre-normalization for stable training.
• —Grouped Query Attention: 4:1 ratio of query to key-value heads.
• —QK Normalization: RMSNorm applied to attention queries and keys.
• —SwiGLU MLP: Gated linear units with SiLU activation.

Usage

Installation

pip install transformers torch

Quick Start

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "Maincode/Maincoder-1B",
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(
    "Maincode/Maincoder-1B",
    trust_remote_code=True,
)

# Code completion example
prompt = '''def fibonacci(n: int) -> int:
    """Return the n-th Fibonacci number."""
'''

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
    **inputs,
    max_new_tokens=256,
    temperature=0.2,
    do_sample=True,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Code Completion

# Function completion
prompt = '''def quicksort(arr: list) -> list:
    """Sort a list using the quicksort algorithm."""
'''

# Class completion
prompt = '''class BinarySearchTree:
    """A binary search tree implementation."""
    
    def __init__(self):
'''

# Algorithm implementation
prompt = '''def dijkstra(graph: dict, start: str, end: str) -> tuple:
    """Find the shortest path using Dijkstra's algorithm.
    
    Args:
        graph: Adjacency list representation of the graph
        start: Starting node
        end: Target node
    
    Returns:
        Tuple of (distance, path)
    """
'''

Additional Notes

Reproducibility

<details> <summary>Model evaluations were run on 8 AMD MI355X GPUs via the <a href="https://github.com/EleutherAI/lm-evaluation-harness">EleutherAI</a> framework.</summary>

docker run --rm -it \
  --device=/dev/kfd --device=/dev/dri --group-add=video \
  --ipc=host --security-opt seccomp=unconfined \
  -v $(pwd):/workspace -w /workspace \
  -e HF_TOKEN \
  -e PYTHONHASHSEED=0 \
  -e TORCH_DETERMINISTIC=1 \
  -e ROCBLAS_ATOMICS_MODE="0" \
  -e MIOPEN_FIND_MODE="1" \
  -e CUBLAS_WORKSPACE_CONFIG=":4096:8" \
  -e HF_ALLOW_CODE_EVAL="1" \
  rocm/pytorch:rocm7.1.1_ubuntu24.04_py3.12_pytorch_release_2.9.1 \
  bash -c 'pip install "lm_eval[hf]" && \
  accelerate launch -m lm_eval \
  --model hf --model_args "pretrained=Maincode/Maincoder-1B,trust_remote_code=True,dtype=float32" \
  --tasks humaneval,humaneval_plus,mbpp_plus,mmlu,gsm8k \
  --device cuda:0 --batch_size 32 --seed 42 \
  --confirm_run_unsafe_code'

</details>

Limitations

• —Context length limited to 2,048 tokens
• —Primarily optimized for Python, performance may vary on other languages
• —May generate code with bugs or security issues - always review generated code

<div style="margin-left:14px; border-left:4px solid #3b82f6; background:rgba(59,130,246,0.08); padding:8px 10px; border-radius:8px; font-size:0.92em; margin:10px 0;"> <strong>Disclaimer</strong>: This model has <strong>not</strong> undergone any alignment or safety tuning (e.g., RLHF/RLAIF, DPO, or safety fine-tuning). Outputs may be unsafe or biased. Please use appropriate safeguards and evaluate carefully for your use case. </div>

License

This model is released under the Apache 2.0 License.

Citation

@misc{maincoder2025,
  title        = {Maincoder-1B: A High-Performance 1B Parameter Coding Model},
  author       = {Maincode Team},
  year         = {2025},
  organization = {Maincode},
  howpublished = {\url{https://huggingface.co/Maincode/Maincoder-1B}}
}

Contact

For questions, issues, or collaboration inquiries, please visit Maincode.