KAS2003/xfield-radar-dataset-20260915

XField radar dataset — formal snapshot, 2026-09-15

Upload status: COMPLETE — every shard verified against its remote SHA-256 and byte size. See `UPLOAD_COMPLETE.json`.

This public research snapshot preserves the currently admitted XField/GRT simulation dataset: radar inputs, existing GT, available raw sensor products, provenance, and the frozen N141 train/validation split. It is not a claim that historical labels meet the newly repaired independent dense-GT pipeline.

Counts and splits

The 532 excluded effective inputs comprise 500 original near-budget exclusions, one unscored exclusion and 31 later experiment-budget exclusions. These are retained records, not newly declared failed samples; they are not training examples. The original 26,129-example supply pool includes validation and must not be used wholesale for fitting.

Use only metadata/train.jsonl for fitting and metadata/validation.jsonl for validation. The 667 training routes and 73 validation routes retain their dependency grouping in metadata/split.json. metadata/records.jsonl is an inventory, not a training list. metadata/not_selected_for_training.jsonl accounts for the remaining 3,502 records, including duplicates.

No new test split is invented. Protected external/model-selection test assets, including locked Town07, are not opened or copied by this upload. Historical Town10 evaluations have selection exposure and must not be called clean unseen testing. Never rename validation as test. Original Town12-0496 is training-exposed.

Included assets and boundaries

Payload selection follows the formal 29,600-record release, not a recursive scan of production:

• —Referenced radar input and GT files, with associated GT metadata.
• —Saved raw sensor files in each referenced sample directory: available radar/LiDAR native data, RGB, BEV, depth, semantic images and metadata.
• —Formal admission receipts, release metadata, frozen split and path aliases.
• —Per-shard and per-file SHA-256 checksums.

Diagnostic captures (including the later strict-document 0496 pilot), research-only HOLDs, rejected/deleted captures, checkpoints, third-party real data and unrelated production files are excluded. Original admitted 0496 remains in its original split. The archived numerical payloads are unchanged. On 2026-09-16, after complete Hub verification and owner authorization, the remote source payloads were removed to make space for a new document-compliant recollection. Scenario, calibration, actor-state and split metadata were retained locally. Restore the payloads from this snapshot before using the historical training paths.

Repository layout

README.md
restore.py
metadata/
  records.jsonl
  train.jsonl
  validation.jsonl
  not_selected_for_training.jsonl
  split.json
  FINAL_PROOF.json
  release.json
  manifest_integrity.json
  file_inventory.jsonl
  source_path_aliases.json
shards/data-00000-of-NNNNN.tar.zst
shard_receipts/data-00000-of-NNNNN.tar.zst.json
UPLOAD_COMPLETE.json

Each independently extractable zstd-compressed tar shard contains approximately at most 4 GiB of source payload. Shards are storage partitions, not train/validation partitions or standard WebDataset samples. A sample may span shards; consult manifests. Extraction creates payload/ and checksums/.

Symlink aliases resolve to one canonical file. Portable input/target/source-metadata fields point to that canonical extracted path. Internal historical JSON may retain machine paths for provenance; it is not an executable restore plan.

Download, verify and load

The dataset is public. Authentication may be used for Hub access; never embed tokens in scripts.

pip install huggingface_hub zstandard
hf auth login

from huggingface_hub import snapshot_download
snapshot_download(
    repo_id="KAS2003/xfield-radar-dataset-20260915",
    repo_type="dataset",
    local_dir="xfield_snapshot",
)

Check UPLOAD_COMPLETE.json reports VERIFIED, then extract and verify:

python xfield_snapshot/restore.py --root xfield_snapshot

The utility verifies compressed shard hashes and all extracted payload checksums. Allow uncompressed payload size plus compressed downloads and filesystem overhead. Archives are not automatically deleted.

from pathlib import Path
import json
import numpy as np
root = Path("xfield_snapshot")
with (root / "metadata/train.jsonl").open() as f:
    sample = json.loads(next(f))
with np.load(root / sample["input"], allow_pickle=False) as radar:
    print(radar.files)
with np.load(root / sample["target"], allow_pickle=False) as target:
    print(target.files)

For training, use maintained XField_PredCore ExperimentRadarDataset and the appropriate producer adapter. This is not a generic image-classification dataset.

Coordinate, input and GT contracts

• —N141 uses native complex I/Q Range–Steering–Doppler input, normally [R512,S25,D64], with physical axes. An RA display image cannot replace it.
• —The output is apparent-space occupancy [A128,R64]. LOS support occupies observed physical locations; reflected support occupies mirror locations. It is not a recovered hidden-object mesh.
• —Preserve target_schema and per-sample metadata. The legacy and explicit producer families are not silently relabeled as newly regenerated supervision.
• —Radar coordinates: x forward, y left, z up. CARLA uses y right. Apply documented calibration, never an arbitrary display flip.
• —Historical projection rules remain unchanged. New-pilot slant-range supervision is not a drop-in relabeling of historical plane-distance targets.
• —Read each saved RGB resolution/FOV and sensor pose. Historical 90-degree captures must not be stretched or relabeled as the Scene 1280×720 / 120-degree contract.
• —Unknown or unobserved space is not proof of free space; preserve producer validity handling.

Known quality limitations

This snapshot supports reproducible historical baselines and supervision rebuilding. It is not a certified new dense-mNLOS release.

The historical pipeline used sparse signal-channel two-bounce hits for much of its mirror supervision. A later isolated repair added independent ultra-dense geometry, native double path records, convergence checks and signal-generation corrections. Those repairs have not been propagated to this whole snapshot. Visible mNLOS energy can coexist with incomplete historical GT; broad model responses do not establish accurate physical recovery.

A 2026-09-15 route-level audit inspected one selected representative frame per route. 715 had explicit actor/sensor poses and motion context; 25 older structures need further investigation. None of those representative raw directories contained independent dense-GT native queries. This is not an all-frame replay or quality certificate.

Ground clutter, material-specific scattering and full real-processing alignment remain incomplete. Do not claim calibrated real SNR, faithful vegetation scattering, or a complete Welch/averaging/background-subtraction pipeline. Existing processed real RA cannot recover missing native I/Q and Doppler.

Future replacements must preserve sample identity and split membership, verify input/GT geometry and producer compatibility, and update manifests, counts, folders and delivery documents together. Matching filenames alone do not justify mixing old and new GT.

Provenance and terms

• —Authority: XField simulation.md.
• —Platform: CARLA with project-specific XField sensor and Python processing.
• —Release: training_release_20260914; split: n141_continue_20260914/splits.
• —Maintainer: KAS2003. Public snapshot explicitly authorized by the dataset owner after private-capacity review.

This upload grants no new public redistribution license. Underlying CARLA/map/assets and project-code terms still apply. No paper DOI or citation is invented.

中文说明

这是当前正式数据的完整公开快照，不代表旧GT已经全部按新文档重建。正式记录29600、有效输入26630；训练23506、验证2592。其余有效排除样本及重复记录保留台账，但不能混入训练。仅按train/validation清单加载。

原始数据、input、GT及来源均保留；诊断、失败、研究HOLD、权重和第三方真实数据不混入。旧GT稀疏监督的问题如实说明，不能仅改README就宣称已修复。上传完成以UPLOAD_COMPLETE.json和分片校验为准。