Shuu12121/NightOwl

NightOwl

NightOwl is a ModernBERT-style code encoder pre-trained from scratch on a diverse mix of source code, natural language, and technical documentation.

NightOwl-large reaches 0.8508 average MRR on MTEB CodeSearchNetRetrieval, exceeding CodeBERT-base (0.7944), GraphCodeBERT-base (0.8325), UniXCoder-base (0.8003), and ModernBERT-base (0.8182) under an identical fine-tuning protocol.

Checkpoints

The NightOwl family is pre-trained in two phases. Both the intermediate (Phase 1) and final (Phase 2) checkpoints are released.

For downstream code search, start from the Phase 2 checkpoints (NightOwl / NightOwl-large).

Model size

Both variants are ModernBERT encoders (alternating local/global attention, RoPE positional embeddings) with a custom 50,368-token BPE tokenizer.

Parameter counts are approximate, derived from the architecture configuration (token embeddings + transformer layers + MLM head).

Training data

NightOwl is trained on two source families. Phase 1 uses every source below; Phase 2 continues on the code-related subsets only.

1. bigcode/starcoder2data-extras (12 subsets)

Diverse code, natural-language, and technical-knowledge subsets. max_samples caps the rows sampled per subset; max_chars truncates very long documents to control memory.

2. Shuu12121/github-file-programs-dataset (8 languages)

Whole-file source code, one Hugging Face repo per language (text field: content). Used in both phases.

python, javascript, typescript, java, go, rust, ruby, php

Sampling caps: Phase 1 — up to 1,000,000 files per language; Phase 2 — up to 2,000,000 files per language.

Training procedure

NightOwl is pre-trained in two phases, both using masked-language modeling with mlm_probability = 0.3.

• —Phase 1 — Mixed pre-training. Standard random-token MLM (mlm collator) over all data sources (code + NL + docs). Produces NightOwl-Pre / NightOwl-large-Pre.
• —Phase 2 — Code-only continuation. Line-level MLM (line_no_space collator) over the code-related subsets only. Entire source-code lines are masked rather than random tokens, aligning the objective with code-search downstream tasks. Produces NightOwl / NightOwl-large.

Long examples are split into chunks (split_long_examples: true) so all tokens are used rather than truncated.

Evaluation

Evaluated on MTEB CodeSearchNetRetrieval after SentenceTransformer fine-tuning on CodeSearchNet pairs (10,000 samples per language, Multiple Negatives Ranking loss). Each model is swept over six learning rates; the best per-model result is reported. Only the pre-trained backbone differs between rows — the fine-tuning and evaluation recipe is held fixed.

CodeSearchNetRetrieval — MRR by language (best across learning rates)

NightOwl-large takes five of seven score columns, including the average.

Usage

NightOwl is an encoder backbone. Load it directly for masked-LM / feature-extraction, or wrap it as a SentenceTransformer for code search.

Masked language modeling

from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("Shuu12121/NightOwl")
model = AutoModelForMaskedLM.from_pretrained("Shuu12121/NightOwl")

Feature extraction

from transformers import AutoTokenizer, AutoModel
import torch

tokenizer = AutoTokenizer.from_pretrained("Shuu12121/NightOwl")
model = AutoModel.from_pretrained("Shuu12121/NightOwl")

code = "def add(a, b):\n    return a + b"
inputs = tokenizer(code, return_tensors="pt", truncation=True, max_length=1024)
with torch.no_grad():
    embeddings = model(**inputs).last_hidden_state  # [1, seq_len, hidden_size]

Limitations

• —NightOwl is an encoder for understanding/retrieval, not a generative model — it does not produce code.
• —Code-search strength is best realized after SentenceTransformer fine-tuning; the raw backbone is not a ready-to-use retriever.
• —Training data is dominated by 8 programming languages; performance on other languages may be lower.
• —Pre-training data is sampled from public sources and may contain bugs, insecure patterns, or biased content.