jonmabe/tiny-llm-cli-sft

Tiny-LLM CLI SFT (54M)

A 54 million parameter language model fine-tuned for CLI command generation.

Model Description

This model is a Supervised Fine-Tuned (SFT) version of jonmabe/tiny-llm-54m, trained to generate Unix/Linux shell commands from natural language instructions.

Training Data

• —Geddy's NL2Bash dataset: ~2,300 natural language to bash command pairs
• —NL2Bash benchmark: Standard benchmark for command translation
• —Synthetic examples: Additional generated pairs
• —Total: ~13,000 training pairs

Training Details

Architecture

• —Parameters: 54.93M
• —Layers: 12
• —Hidden Size: 512
• —Attention Heads: 8
• —Intermediate Size: 1408
• —Max Position: 512
• —Vocabulary: 32,000 tokens
• —Features: RoPE, RMSNorm, SwiGLU, Weight Tying

Usage

Prompt Format

Instruction: <natural language description>
Command:

Example

from model import TinyLLM
import torch

# Load model
checkpoint = torch.load("best_model.pt", map_location="cpu")
model = TinyLLM(checkpoint["config"]["model"])
model.load_state_dict(checkpoint["model_state_dict"])
model.eval()

# Generate
prompt = "Instruction: Find all Python files modified in the last day\nCommand:"
# ... tokenize and generate

Limitations

⚠️ Known Issues:

• —Tokenizer decode shows raw BPE tokens (Ġ = space, Ċ = newline)
• —Model generates fragments of correct commands but output can be noisy
• —Needs more training steps for reliable generation
• —Small model size limits command complexity

Improvement Plan

1. 1.Fix tokenizer decode - Proper BPE to text conversion
2. 2.Longer training - 5,000-10,000 steps
3. 3.Data quality - Curate cleaner training pairs
4. 4.Lower LR - More stable convergence with 1e-5

License

Apache 2.0

Citation

@misc{tiny-llm-cli-sft-2026,
  author = {Jon Mabe},
  title = {Tiny-LLM CLI SFT: Small Language Model for Command Generation},
  year = {2026},
  publisher = {HuggingFace},
  url = {https://huggingface.co/jonmabe/tiny-llm-cli-sft}
}