tegridydev/opensec-triage

opensec-triage 0.5.0

Synthetic English security alert data for disposition classification, counterfactual evaluation and small model training experiments.

Each example pairs a security observation with contextual evidence and an expected disposition. The task is to classify the supplied evidence rather than infer a disposition from the observable action alone.

Configurations

The default configuration remains the default dataset view and loading target. Release 0.5.0 adds the revised edge_trainer 0.2.0 curriculum and its aligned chat view. It does not change the existing main or model_trainer data files. Research and chat views are alternate representations, not additional independent examples.

Default dataset

The main dataset contains 50,000 examples in 10,000 counterfactual groups. Each group shares the same primary observation, asset and identity while varying the evidence and disposition.

Splits

Family profiles, counterfactual groups, assets and identities are disjoint across the main splits.

Labels

model_trainer

model_trainer is an additional 4,200 row curriculum designed for compact models that need sharper separation between nearby dispositions.

It targets these boundaries:

• —suspicious vs unknown
• —benign vs expected_admin
• —authorised_testing vs expected_admin
• —misconfiguration vs malicious
• —malicious vs suspicious

Every counterfactual group contains seven records with the same observable event, asset, principal and operating context. Only the decision evidence changes, producing one record for each disposition. This prevents the event type itself from being a reliable shortcut.

model_trainer splits

Behaviour families and counterfactual groups are disjoint across these splits. Validation and test use wording held out from the training evidence bank. Evidence sentence order is varied deterministically.

The model_trainer training file contains only text and target. The aligned metadata files are for analysis and must not be included as model inputs.

edge_trainer 0.2.0

edge_trainer contains compact synthetic event and evidence bundles for small encoders and causal language models. It uses the same seven dispositions and keeps the primary text and target interface.

All twelve domains appear in each split. Each domain contributes eight behaviour IDs to training, one to validation and one to test. Behaviour IDs, observation groups and generated identities do not cross the edge splits. Evidence clause templates are separate between these splits, although conceptual rules and vocabulary remain shared.

Each observation is paired with seven alternative evidence states, one per disposition. Concrete routine workflows and reproducible settings faults accompany the 120 observable patterns. Cases include mismatched approvals, out of scope test claims, explained anomalies and incomplete attribution. The provided evidence, not a risky looking event name, determines the label.

Input representations include parseable normalised JSON, key/value text, syslog-like text and event briefs. They are synthetic normalised records, not vendor parser fixtures. The measured primary inputs span 210 to 394 characters across all splits. Character lengths are not token counts; use the included tokenizer audit before reducing the context window.

Chat view

edge_trainer_chat contains messages with a fixed system instruction, the user event text, and the exact assistant label. It has identical row ordering and split membership to edge_trainer. Do not count or merge it as independent data.

For models trained with this chat view, use the exact instruction in `edge_trainer/prompts/classifier_system.txt` during inference as well. Encoder classifiers consume only text without a system message. Apply the chosen model's chat template once and verify the assistant label survives tokenisation and loss masking.

Intended scope

These are enriched event bundles, not raw log threat detection ground truth. A deployed device must receive the referenced approval, baseline and correlation evidence through its surrounding system. It must not invent those facts from a standalone log line.

The curriculum remains strongly templated. A simple word TF-IDF and logistic regression sanity check scored 100% validation accuracy in this build, demonstrating that lexical rules can solve this synthetic task. This is not proof of operational security reasoning. Independent analyst reviewed data is needed for deployment evaluation.

Full design, limits, schemas and tools are documented in `edge_trainer/README.md`. The subset's 28 additional authored boundary checks are separate from its 8,400 core records and are not included in the Hugging Face training splits.

Loading

from datasets import load_dataset

# Main dataset
main = load_dataset("tegridydev/opensec-triage")

# Targeted compact model curriculum
trainer = load_dataset("tegridydev/opensec-triage", "model_trainer")

# Compact edge classification view
edge = load_dataset("tegridydev/opensec-triage", "edge_trainer")

# Same edge rows as conversations for supervised fine tuning
edge_chat = load_dataset("tegridydev/opensec-triage", "edge_trainer_chat")

# Main research representation
research = load_dataset("tegridydev/opensec-triage", "research")

For supervised fine tuning, use text as the user input and target as the assistant completion or classification target.

Data files

Composition and provenance

The main catalogs contain 100 behaviour definitions, 100 family profiles and 100 evidence variants. Examples combine authored observations, business contexts and evidence patterns.

The model_trainer subset is separately authored synthetic curriculum data. It contains 100 observable behaviour families and 600 counterfactual groups. It was designed to make nearby disposition boundaries explicit while keeping the observable action non decisive.

All records in every configuration are synthetic. No production incident corpus was ingested. Asset and principal identifiers are synthetic placeholders. Investigation guidance and decision policies are authored context rather than collected telemetry.

Training and evaluation

Use text as the model input and target as the label. Do not add research fields such as variant IDs, evidence rules, rationale, nearest confusers or counterfactual group IDs to the model input because they can expose target structure.

For the main dataset, report macro F1, per class precision and recall, confusion matrices and consistency across related examples. Check tokenisation so evidence at the end of each text is retained.

For models trained with model_trainer, evaluate on both:

1. 1.model_trainer/test to measure generalisation across held out behaviour families and evidence wording.
2. 2.default/test to measure compatibility with the original dataset distribution.

Use independently phrased analyst reviewed cases as a separate out of distribution benchmark.

Limitations

The main dataset shares evidence templates and constituent business contexts across its splits. Template memorisation can therefore inflate test scores. Unique text and held out identities do not establish semantic independence.

The model_trainer configuration reduces that specific issue by holding out behaviour families and evidence wording, but it is still synthetic curriculum data and does not establish real world security performance.

Labels follow authored evidence rules rather than independent incident adjudication. Some combinations of observations and workflows require domain review.

Disposition labels do not provide calibrated severity, confidence, likelihood or difficulty estimates. Business context and timing alone do not establish authorisation.

Assess operational suitability separately with analyst reviewed cases and independently phrased evidence.

Edge training and evaluation

Use edge_trainer/train for encoder training or edge_trainer_chat/train for conversational supervised fine tuning. Choose checkpoints on edge_trainer/validation and reserve edge_trainer/test for the final selected checkpoint. Chat model evaluation uses raw event text from the primary view plus the same system instruction as training; never expose the expected assistant answer from test conversations.

Keep exact output accuracy, macro F1, per class recall, invalid outputs and runtime errors separate. A fixed classification head prevents arbitrary output strings, not incorrect predictions. Record dataset revision, tokenizer, chat template, checkpoint, precision and runtime alongside scores.

Cross-configuration semantic overlap has not been eliminated. Parent dataset tests are transfer probes, not guaranteed independent incident evaluations. The 0.2.0 edge split assignments and wording replace 0.1.0; compare models on the same version rather than comparing scores across changed tests.

Exact tokenizer counts were not measured in the build environment. edge_trainer/tools/validate.py --tokenizers measures the actual input and full conversation lengths without downloading model weights. Do not infer a safe training context from a characters-per-token estimate.

Release provenance

The edge 0.2.0 records use AI-assisted synthetic authoring and deterministic assembly with seed 3407. All generated identities are placeholders. This update introduces no real credentials, live network targets or collected incidents. The main dataset and model trainer provenance statements above remain unchanged.

Licence

Licensed under CC BY 4.0.