Shuu12121/CodeModernBERT-Finch
CodeModernBERT-Finch
This model is a code-specific pretrained model created solely using the CodeSearchNet dataset. It supports six languages included in CodeSearchNet.\ For a version fine-tuned specifically for code search tasks, please refer to Shuu12121/CodeSearch-ModernBERT-Finch.
Architecture
- Base: ModernBERT-style encoder
- Hidden size: 512
- Layers: 6
- Attention heads: 6
- Parameters: \~50M
- Pretraining: Masked Language Modeling (MLM)
- Fine-tuning: Domain-specific code tasks
The results below were obtained by randomly sampling 10,000 examples per language from the CodeSearchNet dataset, training them in a Sentence-BERT fashion, and evaluating on the MTEB CodeSearchNetRetrieval benchmark. All models listed in the table below were fine-tuned using the same approach. Those marked with 200 and the Finch models were trained with a Multiple Negatives Ranking Loss batch size of 200. Others were trained with a batch size of 40 (because larger batches could not fit into memory).\ Finch-SmallBatch was trained with a smaller batch size of 40 to create a comparison model against the standard Finch models trained with batch size 200.