eduard76/mistral7b-netops-teamleader

mistral3b-netops-teamleader

Note on repo name: This repo is named mistral3b-... but the actual base model is Mistral-7B-Instruct-v0.3 (4096 hidden size, 32 layers). The 3b in the name is a misnomer from early project planning. The weights are 7B.

Model Description

Fine-tuned mistralai/Mistral-7B-Instruct-v0.3 for Cisco IOS syslog classification into TROUBLESHOOTING / STABILITY / SECURITY for the VexpertAI agentic network operations platform.

This model powers the Team Leader agent log-routing decision: given a raw syslog line from a Cisco IOS router, it emits exactly one of three routing labels that determines which specialist agent handles the incident.

Base Model

mistralai/Mistral-7B-Instruct-v0.3

• —Architecture: MistralForCausalLM
• —Parameters: 7B (hiddensize=4096, numhidden_layers=32)

Training Data

• —Type: Synthetic, IOS syslog format
• —Size: 2,100 labeled examples (700 per class) + 150 hard-negative samples
• —Lab topology: EVE-NG digital twin — two Cisco IOS routers (R1, R2) running OSPF area 0
• —Syslog format: *MMM DD HH:MM:SS.mmm: %FACILITY-SEVERITY-MNEMONIC: message
• —Classes:
• —TROUBLESHOOTING: connectivity loss, interface flaps, adjacency changes, reachability failures
• —STABILITY: CPU/memory exhaustion, process crashes, hardware errors, recurring instability
• —SECURITY: ACL deny hits, login failures, brute force, SNMP auth failures, IPS signatures

Evaluation Results

Inference Example

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "eduard76/mistral3b-netops-teamleader"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

syslog = "*Mar  1 00:05:23.123: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/0, changed state to down"

messages = [
    {"role": "system", "content": "You are a network operations classifier. Given a Cisco IOS syslog line, output exactly one word: TROUBLESHOOTING, STABILITY, or SECURITY. No explanation."},
    {"role": "user", "content": syslog},
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt")

# IMPORTANT: use max_new_tokens >= 12, not 5.
# "TROUBLESHOOTING" tokenizes to 7 tokens — max_new_tokens=5 silently truncates it
# and produces an INVALID label.
output = model.generate(**inputs, max_new_tokens=12, do_sample=False)
new_tokens = output[0][inputs["input_ids"].shape[1]:]
print(tokenizer.decode(new_tokens, skip_special_tokens=True).strip().split()[0].upper())
# → TROUBLESHOOTING

Intended Use

• —Primary use: Team Leader agent in VexpertAI agentic NetOps platform for routing syslog events to the correct specialist agent (Stability, Troubleshooting, or Security)
• —Secondary use: Offline syslog triage, NOC alert pre-classification

Out of Scope

• —Not for direct network configuration changes
• —Not production-validated on live Cisco IOS traffic (trained on synthetic data)
• —Not a replacement for human network engineers in critical incidents
• —Not tested on NX-OS, IOS-XE, or IOS-XR syntax variants

Training Details

• —LoRA: r=16, alpha=32, dropout=0.05, target_modules: q/k/v/o/gate/up/down projections
• —Quantization: 4-bit NF4 QLoRA (bitsandbytes)
• —Epochs: 3
• —Effective batch size: 16 (batch=4, grad_accum=4)
• —Learning rate: 2e-4 cosine schedule, warmup_ratio=0.05

Reproduction Gotchas

These bugs were hit during the training pipeline on an AWS EC2 g4dn.xlarge (15GB RAM, T4 GPU):

1. 1.`max_new_tokens` truncation — Using max_new_tokens=5 in the evaluation loop caused all TROUBLESHOOTING predictions to be marked INVALID (F1=0.0), because that label tokenizes to 7 tokens in Mistral's tokenizer. Use max_new_tokens >= 12 to safely cover all three class labels.

1. 1.OOM on LoRA merge — Loading the 7B base model in FP16 for merge_and_unload() peaks at ~28GB (two copies of the model in memory). On a 15GB RAM instance with no swap this kills the process with exit code -9. Fix: add 20GB swap before running the merge step:

   sudo fallocate -l 20G /swapfile && sudo chmod 600 /swapfile
   sudo mkswap /swapfile && sudo swapon /swapfile

1. 1.Deprecated HuggingFace kwargs — push_to_hub(use_auth_token=..., safe_serialization=...) raises TypeError on current transformers/huggingface_hub versions. Use huggingface_hub.HfApi().upload_folder() instead, which is stable and streams directly from disk without reloading the model into memory:

   from huggingface_hub import HfApi
   HfApi().upload_folder(folder_path="final_model/", repo_id="your/repo", token=hf_token)