codechrl/bert-micro-cybersecurity

1---2language:3- en4- id5tags:6- bert7- text-classification8- token-classification9- cybersecurity10- fill-mask11- named-entity-recognition12- transformers13- tensorflow14- pytorch15- masked-language-modeling16base_model: boltuix/bert-micro17library_name: transformers18pipeline_tag: fill-mask19---20# bert-micro-cybersecurity21 22## 1. Model Details23**Model description**  24"bert-micro-cybersecurity" is a compact transformer model adapted for cybersecurity text classification tasks (e.g., threat detection, incident reports, malicious vs benign content).25- Model type: fine-tuned lightweight BERT variant  26- Languages: English & Indonesia27- Finetuned from: `boltuix/bert-micro`28- Status: **Early version** — trained on **47.53%** of planned data.29 30**Model sources**  31- Base model: [boltuix/bert-micro](https://huggingface.co/boltuix/bert-micro)32- Data: Cybersecurity Data33 34## 2. Uses35### Direct use  36You can use this model to classify cybersecurity-related text — for example, whether a given message, report or log entry indicates malicious intent, abnormal behaviour, or threat presence.37### Downstream use  38- Embedding extraction for clustering.39- Named Entity Recognition on log or security data.40- Classification of security data.41- Anomaly detection in security logs.  42- As part of a pipeline for phishing detection, malicious email filtering, incident triage.  43- As a feature extractor feeding a downstream system (e.g., alert-generation, SOC dashboard).44### Out-of-scope use  45- Not meant for high-stakes automated blocking decisions without human review.  46- Not optimized for languages other than English and Indonesian.  47- Not tested for non-cybersecurity domains or out-of-distribution data.48 49### Downstream Usecase in Development using this model50- NER on security log, botnet data, and json data.51- Early classification of SIEM alert & events.52 53## 3. Bias, Risks, and Limitations 54Because the model is based on a small subset (47.53%) of planned data, performance is preliminary and may degrade on unseen or specialized domains (industrial control, IoT logs, foreign language).55- Inherits any biases present in the base model (`boltuix/bert-micro`) and in the fine-tuning data — e.g., over-representation of certain threat types, vendor or tooling-specific vocabulary.56- **Should not be used as sole authority for incident decisions; only as an aid to human analysts.**57 58## 4. Training Details59 60### Text Processing & Chunking61Since cybersecurity data often contains lengthy alert descriptions and execution logs that exceed BERT's 512 token limit, we implement an overlapping chunking strategy:62- **Max sequence length**: 512 tokens63- **Stride**: 32 tokens (overlap between consecutive chunks)64- **Chunking behavior**: Long texts are split into overlapping segments. For example, with max_length=512 and stride=128, a 1000-token document becomes ~3 chunks with 128-token overlaps, preserving context across boundaries.65 66### Training Hyperparameters67- **Base model**: `boltuix/bert-micro`68- **Training epochs**: 369- **Learning rate**: 5e-0570- **Batch size**: 1671- **Weight decay**: 0.0172- **Warmup ratio**: 0.0673- **Gradient accumulation steps**: 174- **Optimizer**: AdamW75- **LR scheduler**: Linear with warmup76 77### Training Data78- **Total database rows**: 348,72279- **Rows processed (cumulative)**: 165,761 (47.53%)80- **Training date**: 2026-04-06 03:08:3081 82### Post-Training Metrics83- **Final training loss**: 84- **Rows→Samples ratio**: 85