thealper2/lfm2-700m-linux-command

thealper2/lfm2-700m-linux-command

LiquidAI/LFM2-700M fine-tuned to map a natural-language Linux task to a single shell command. The target output is the command only; no explanation is produced.

Model details

Prompt format

The model uses the LFM2 ChatML-style chat template and was trained with no system prompt. Apply the template rather than constructing the string manually.

<|startoftext|><|im_start|>user
Find which process is using port 8080.<|im_end|>
<|im_start|>assistant
lsof -i :8080<|im_end|>

Two LFM2 tokenizer details matter:

• —The chat template emits bos_token itself and add_bos_token is true in tokenizer_config.json. Tokenise templated text with add_special_tokens=False, or the sequence gets a duplicated BOS.
• —Decode with clean_up_tokenization_spaces=False; the BPE cleanup step strips spaces around punctuation and can corrupt shell commands.

Special tokens: BOS <|startoftext|> (1), EOS <|im_end|> (7), PAD <|pad|> (0).

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "thealper2/lfm2-700m-linux-command"
tokenizer = AutoTokenizer.from_pretrained(model_id, clean_up_tokenization_spaces=False)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16).to("cuda").eval()

messages = [{"role": "user", "content": "Find which process is using port 8080."}]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)

output = model.generate(
    **inputs,
    max_new_tokens=128,
    do_sample=False,                      # deterministic decoding
    eos_token_id=tokenizer.eos_token_id,
    pad_token_id=tokenizer.pad_token_id,
)
command = tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip()
print(command)  # lsof -i :8080

Greedy decoding (do_sample=False) is the intended configuration: the task has a single intended answer and sampling only adds variance.

Training data

Both were normalised to {instruction, command, source} and then:

1. 1.Cleaned — markdown fences and $/# prompt prefixes stripped, whitespace runs collapsed outside quoted strings, records with unbalanced quotes, prose instead of a command, or no parseable utility dropped. Commands themselves were never rewritten.
2. 2.Deduplicated — exact (instruction, command) duplicates removed. Rows sharing a command with a different instruction, or an instruction with a different valid command, were kept deliberately.
3. 3.Balanced — find was 35.2% of the raw corpus. It was capped per-utility using a diversity-aware ordering that retains every distinct flag signature before retaining any repeat, lowering its share to ~19%.
4. 4.Split — 90/5/5, grouped by instruction template (filenames, paths, numbers and quoted literals abstracted) so templated paraphrases cannot straddle the train/test boundary.

Split sizes: 11756 train / 652 validation / 653 test. Leakage checks report zero overlap across splits at the exact-pair, instruction and instruction-template level.

Training configuration

Loss is computed on the assistant turn only; prompt tokens are masked with -100.

max_length was chosen from the tokenised length distribution of the corpus (mean 37.7, median 34, p90 57, p95 66, p99 87, max 403) — a 128-token budget covers 99.94% of examples.

Run record

Evaluation

Measured on the held-out test set with greedy decoding.

Metric definitions:

• —Exact match — string equality after stripping surrounding whitespace.
• —Normalised exact match — equality after collapsing whitespace runs outside quotes and removing a trailing ;.
• —Structural match — utility, flag multiset (short-flag bundles expanded for utilities that use them) and operand sequence compared per pipeline segment. Recognises ls -la == ls -al.
• —Command validity — the output parses under a bash grammar parser (bashlex), not membership in a list of known utilities.
• —Token F1 — token-level overlap, as partial credit.
• —Primary-utility accuracy — the first utility matches the reference.
• —Prose-output rate — fraction of outputs that read as an explanation rather than a command.

Limitations

• —Structural match is not a semantic oracle. It compares command shape. It cannot tell that find . -name '*.py' and a shell glob achieve the same result, and it does not reason about flag semantics. It is an upper bound on exact match, not semantic accuracy.
• —Exact match understates correctness. Many Linux tasks have several valid answers; the test set carries one reference each.
• —Source-distribution bias. nl2bash is find-heavy and composition-heavy; mecha-org/linux-command-dataset is templated and single-utility-heavy. Per-source metrics differ and are reported separately in the project reports.
• —Distribution shift. Commands reference paths, hosts and variables that appear in the training corpora (/path/to/..., $source). Outputs may embed those placeholders instead of the user's real paths.
• —Short outputs only. Trained at a 128-token budget; long multi-stage scripts are out of distribution.
• —No verification of correctness or safety at generation time. The model can produce syntactically valid but wrong — or destructive — commands.

Intended use and safety

Intended for generating candidate shell commands for review, and as the command generator of a sandboxed terminal agent.

Do not execute generated commands directly on a host. The project this model comes from executes commands only inside a disposable Docker container started with --network none, --read-only, --cap-drop ALL, --security-opt no-new-privileges, a non-root user, no host bind mounts, bounded CPU/memory/PIDs and a wall-clock timeout, and screens commands against a destructive-pattern list and a read-only allowlist before running them.

License

Inherits the LFM Open License v1.0 of the base model, LiquidAI/LFM2-700M. Dataset licenses apply to the training data: mecha-org/linux-command-dataset is Apache-2.0; nl2bash derives from the TellinaTool/nl2bash corpus.

Citation

The NL2Bash corpus:

@inproceedings{LinWZE2018:NL2Bash,
  author    = {Xi Victoria Lin and Chenglong Wang and Luke Zettlemoyer and Michael D. Ernst},
  title     = {NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating System},
  booktitle = {LREC 2018},
  year      = {2018}
}