nraptisss/TMF921-intent-to-config-research-sota

TMF921 Intent-to-Config Research SOTA Splits

This dataset is a research-oriented derivative of `nraptisss/TMF921-intent-to-config-augmented`. It provides reproducible training and OOD evaluation splits for supervised fine-tuning models that translate natural-language telecom/network-slicing intents into structured JSON configuration objects.

This dataset is intended for research. It is not a production-certified telecom configuration generator and should not be used to deploy network configurations without expert review and standards-specific validation.

Why this derivative exists

A scientific audit of the source dataset found that the data is technically clean and SFT-ready, but the original train/test split is primarily in-distribution and template-like. This derivative implements the audit recommendations that can be implemented safely from the released parquet files without fabricating unsupported target JSON.

Key audit findings on the source dataset:

• —41,815 total rows.
• —100% assistant JSON parse validity.
• —0 missing values.
• —0 duplicate IDs.
• —0 exact train/test full-message overlaps.
• —high near-duplicate prompt similarity across the original split.
• —strong lifecycle imbalance: create operations dominate.
• —adversarial rows are rare.

Dataset construction

This derivative was produced from the source dataset by:

1. 1.merging the source train/test rows,
2. 2.adding derived research/provenance columns,
3. 3.constructing explicit OOD evaluation splits,
4. 4.adding a token-length audit using Qwen/Qwen3-8B,
5. 5.adding validation flags,
6. 6.creating two training splits:
7. 7.train_base: unaugmented after OOD holdouts,
8. 8.train_sota: training split with marked lifecycle/adversarial upsampling and synthetic multi-turn wrappers.

No new continuous-KPI target configurations or fake cross-layer paired 6-tuples were fabricated. Safe generation of those variants requires a canonical scenario generator plus per-layer schema validators.

Splits

Evaluation results should be reported per split. Do not merge OOD splits into a single score without stratified reporting.

Columns

Original core columns include:

• —id
• —messages
• —target_layer
• —slice_type
• —sst
• —sd
• —use_case
• —sector
• —region
• —latency_ms
• —reliability_pct
• —dl_throughput_mbps
• —ul_throughput_mbps
• —max_ues
• —lifecycle_operation

Added research columns include:

• —system, prompt, completion: extracted SFT convenience fields.
• —prompt_template_id: normalized prompt-family hash for leakage-aware splitting.
• —scenario_id: metadata scenario hash.
• —json_structure_id: assistant JSON structure hash.
• —json_root_family: top-level target family.
• —messages_format_valid: whether message format is structurally valid.
• —assistant_is_valid_json: whether assistant output parses as JSON.
• —slice_sst_valid: slice/SST consistency check.
• —kpi_profile_valid: KPI range validity against dataset profiles.
• —semantic_rule_valid_v1: combined lightweight semantic validity flag.
• —qwen3_chat_template_tokens: token length using Qwen/Qwen3-8B chat template.
• —fits_2048_qwen3, fits_4096_qwen3: token-length flags.
• —sampling_weight_region_balanced, sampling_weight_lifecycle_balanced, sampling_weight_recommended: optional sampling weights.
• —is_augmented, augmentation_type, source_id, conversation_type: augmentation/provenance columns.
• —research_split: assigned research split.

Token-length audit

The source rows were audited with the Qwen/Qwen3-8B chat template.

{
  "count": 41815,
  "mean": 754.1127107497309,
  "min": 120,
  "p50": 705,
  "p90": 1288,
  "p95": 1293,
  "p99": 1300,
  "max": 1316
}

All audited source rows fit within 2048 tokens for Qwen3. max_length=2048 is therefore a justified default for Qwen3-family SFT. Researchers using another tokenizer should recompute token lengths.

Recommended use

• —Reproducible baseline training: use train_base.
• —Stronger rare-class learning: use train_sota.
• —Model selection/validation: use validation.
• —Scientific evaluation: report all test splits separately.
• —Use prompt_template_id, scenario_id, and split names for leakage-aware experiments.

Recommended evaluation protocol

At minimum, report:

1. 1.JSON parse rate.
2. 2.Raw field precision/recall/F1.
3. 3.Normalized field precision/recall/F1.
4. 4.Normalized key precision/recall/F1.
5. 5.Metrics per target layer.
6. 6.Metrics per OOD split.
7. 7.Adversarial rejection accuracy.
8. 8.Qualitative failure examples.

Raw exact match is not sufficient because many valid outputs contain volatile fields such as generated IDs, hrefs, timestamps, descriptions, and schema links.

Known limitations

• —The dataset is synthetic and controlled; it is not a corpus of real operator logs.
• —OOD splits are controlled synthetic OOD splits, not real-world deployment distributions.
• —This derivative does not include official TMF921/3GPP/ETSI/CAMARA/O-RAN validators.
• —semantic_rule_valid_v1 is a lightweight dataset-level rule check, not standards certification.
• —O1 NRM and A1 policy value fidelity remain challenging for current models.
• —Some lifecycle monitor/report rows may include simulated measurements that require tolerance-based or semantic scoring.
• —Continuous-KPI variants and cross-layer paired 6-tuples are not generated in this derivative because doing so safely requires canonical scenario renderers and validators.

Intended use

This dataset is intended for:

• —supervised fine-tuning research,
• —structured JSON generation research,
• —telecom LLM benchmarking,
• —OOD evaluation method development,
• —intent-based networking experiments,
• —comparison of open-weight models on telecom configuration generation.

Out-of-scope use

This dataset and models trained on it should not be used directly to:

• —deploy real network configurations,
• —alter production telecom systems,
• —bypass operator review,
• —claim official standards compliance without validators,
• —make safety-critical network-management decisions.

Ethical and security considerations

Generated network configurations can be operationally sensitive if applied to real systems. This dataset contains synthetic examples and no real credentials, secrets, or operator network data. Nevertheless, models trained on it should be treated as decision-support tools requiring human/operator review and schema validation before any operational use.

Adversarial robustness is preliminary. The adversarial subset is useful for research but does not cover the full range of prompt-injection, malicious, conflicting, or unsafe network-management requests.

Artifacts

• —artifacts/split_statistics.json
• —artifacts/validation_summary.json
• —artifacts/research_split_manifest.json
• —artifacts/gen4_generator_blueprint.json
• —artifacts/post_build_validation.json

Related code and results

Training, evaluation, normalized scoring, results, qualitative examples, and project journal are available at:

https://huggingface.co/nraptisss/tmf921-intent-training

Citation

If you use this dataset, cite the dataset repository and the associated training/evaluation repository:

@dataset{raptis_tmf921_research_sota_2026,
  title = {TMF921 Intent-to-Config Research SOTA Splits},
  author = {Raptis, Nikolaos},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/datasets/nraptisss/TMF921-intent-to-config-research-sota}
}

License

Apache-2.0, following the source dataset.