Skyl4r-Ai/Skylar-390M-Cobol

Skylar-390M-Cobol

A 390M-parameter, from-scratch language model specialized in COBOL — small enough to run locally and on-prem, trained entirely in-house, that outperforms 7B general code models on COBOL code generation.

🔗 Watch it grow: https://skyl4r.ai

<img src="https://raw.githubusercontent.com/skyl4r-ai/skylar/refs/heads/main/docs/skylar-cobol-demo.gif" alt="A from-scratch Skylar model generates COBOL; GnuCOBOL compiles and runs it" width="760"/>

_{A from-scratch Skylar model completes a COBOL task → GnuCOBOL compiles it → it runs → correct output. No internet, no API, no third-party weights.}

TL;DR

• —~390M parameters (386M exact), decoder-only, Qwen3-style architecture, trained 100% from scratch (no warm-start from anyone else's weights).
• —Domain: COBOL — the language that still runs banking, insurance and public-administration back-ends, where general code models perform poorly and skilled developers are retiring.
• —Sovereign: runs on a single local GPU, no data leaves your infrastructure, no dependency on any external LLM provider.
• —On COBOLEval (146 HumanEval-derived COBOL problems, compiled and executed with GnuCOBOL), it beats Qwen2.5-Coder-7B, CodeLlama-7B and StarCoder2-7B on pass@1 — at 18× fewer parameters.

Results — head-to-head on COBOLEval

All models evaluated on the same harness, greedy decoding, seed 0, GnuCOBOL compile+execute, official {NAME}.TXT scoring.

Ordered by pass@1 (the benchmark). Skylar-390M-Cobol is #1 among all deployable models — only the much larger reference teacher scores higher.

[image]

_{pass@1 = 8/146 problems solved. 95% binomial CI ≈ [2.4%, 10.5%] — low absolute, as expected for a 390M model on an execution benchmark; the point is the relative result vs the 7B baselines on the identical harness.}

Why this comparison is trustworthy: our measured baseline scores match the published literature (StarCoder2 ~0%, CodeLlama ~0%, Qwen2.5-Coder ~0.68% on COBOLEval). The harness does not favor us — it reproduces the competitors' official numbers, and on that same harness the specialized 390M wins.

What the numbers mean. The general 7B models often write pseudo-COBOL — they call mod(), substr(), abs() (functions from Python/Java that don't exist in COBOL), so their code looks right but doesn't compile. Skylar-390M-Cobol writes real COBOL: it compiles 82% of the time and solves the most problems. This is the value of specialization.

Decontamination

We audited overlap between training data and the COBOLEval test set. 4 problems (below_zero, count_upper, find_max, is_sorted) appeared in both; we removed them from the training set and retrained so the released model has never seen any test problem. The numbers above (5.5% / 82.2%) are reported on this clean model — and as a sanity check, none of the 4 formerly-overlapping problems are among the solved ones, confirming the score carries no leakage.

Intended use & limitations

Intended use: a supervised assistant for COBOL developers — scaffolding, boilerplate, DIVISION/LINKAGE structure, syntax, completion — where a human reviews the output.

Limitations (read this):

• —pass@1 is 5.5% in absolute terms — low. COBOLEval is hard (real execution), but this model is not capable of autonomously writing correct COBOL programs end-to-end. It is capacity- limited: it compiles well (82%) but its algorithmic reasoning is weak.
• —Do not use it unsupervised or for safety-critical code.
• —English/COBOL only; not a general chat model.

⚠️ Research preview (v0.1). This is an honest proof-of-concept, not a production tool. It demonstrates that a small, sovereign, specialized model can beat much larger generalist code LLMs on a niche legacy language. Read the Limitations before using it. Numbers are reported with full methodology and a decontamination audit — no hidden asterisks.

How it was built

1. 1.Pretrain (from scratch): 8.19B tokens, mix of permissive code (oversampled), Italian/EU legal text, and COBOL, Chinchilla-optimal, code-aware BPE tokenizer (vocab 32768).
2. 2.SFT: instruction tuning on COBOL completion, with verified distillation — a strong teacher solves COBOL problems, each solution is kept only if it compiles and passes its tests, then used as training data. Plus targeted synthetic data for structural gaps.
3. 3.Decontamination + clean retrain (see above).

Roadmap

This 390M is the first step. A larger model (~1B) with more code pretraining and longer context is in progress — that is the version intended to become a genuinely usable COBOL assistant.

How to use it — a COBOL completer, not a chatbot

Skylar-390M-Cobol completes COBOL; it does not chat. You give it a COBOL stub — a skeleton with the task written as comments, ending at WORKING-STORAGE SECTION. — and it writes the WORKING-STORAGE entries + PROCEDURE DIVISION. Do not prompt it with free-form natural language ("write me a program that…", and especially not in Italian): it will ramble, not code.

pip install skylar
skylar cobol --example                      # the built-in demo stub
skylar cobol --stub-file my_task.cbl --compile   # your own stub → completed + GnuCOBOL check

import skylar
m = skylar.load("Skyl4r-Ai/Skylar-390M-Cobol")
print(m.complete_cobol(my_stub))            # reassembles a full, compilable program

✅ Tasks it handles reliably — simple list / loop / accumulate / string work, e.g. max_element, sum_to_n, strlen, incr_list, is_palindrome, flip_case, pairs_sum_to_zero, count_up_to. Give it a stub shaped like these.

✅ Worked example — stub in → compilable program out (real output, cobc accepts it):

*> INPUT STUB (ends at WORKING-STORAGE SECTION.):
       01 LINKED-ITEMS.
           05 L-L OCCURS 3 TIMES INDEXED BY NI PIC S9(10).
           05 RESULT PIC S9(10).
      * Return maximum element in the list.
      * >>> max_element([5, 3, -5])  ->  5
       WORKING-STORAGE SECTION.

*> Skylar-390M-Cobol COMPLETES it:
       01 WS-I PIC 9(4).
       PROCEDURE DIVISION USING LINKED-ITEMS.
           MOVE L-L(1) TO RESULT
           PERFORM VARYING WS-I FROM 2 BY 1 UNTIL WS-I > 3
               IF L-L(WS-I) > RESULT  MOVE L-L(WS-I) TO RESULT  END-IF
           END-PERFORM
           GOBACK.
       END PROGRAM MAX-ELEMENT.

⚠️ Secondary mode: a precise technical English instruction also works, e.g. skylar generate --prompt "Write a COBOL PROCEDURE DIVISION that sums WS-NUMBERS into WS-TOTAL and displays it." — but the stub format above is where it's strongest.

❌ Don't: chat with it, prompt in natural Italian, or expect correct programs for hard tasks (pass@1 is 5.5% — it compiles ~82% of the time but its algorithmic reasoning is weak). Always review the output.

import skylar also registers the architecture with 🤗 Transformers, so AutoModelForCausalLM.from_pretrained("Skyl4r-Ai/Skylar-390M-Cobol") works too. The weights (model.safetensors) and tokenizer.json are standard; the architecture is a custom decoder (NanoTransformer, model_type: nano-transformer — Qwen3-style RMSNorm + RoPE + GQA + QK-Norm + SwiGLU), described in full in the accompanying paper.

License & attribution

Apache-2.0. IP: A. Ivanovitch (CEO of Skylar Project). Built in-house, no third-party pretrained weights.