mehmetkeremturkcan/SyntheticStreetScenes

Synthetic Street Scenes

<p align="center"> <a href="https://keremturkcan.com/">Mehmet Kerem Turkcan</a> <br> Columbia University, Center for Smart Streetscapes (CS3) <br><br> <a href="https://huggingface.co/mehmetkeremturkcan/StreetPrompt"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Model-blue" alt="StreetPrompt model"></a> <a href="https://creativecommons.org/licenses/by/4.0/"><img src="https://img.shields.io/badge/License-CC%20BY%204.0-lightgrey.svg" alt="License CC BY 4.0"></a> </p>

<p align="center"> <a href="assets/overview.jpg"><img src="assets/overview.jpg" width="100%" alt="One frame from each of the 22 datasets, grouped by viewpoint: traffic and street cameras, roadside sensors and fixed municipal cameras, egocentric and low fixed cameras, and the original humanoid robot viewpoint"></a> </p>

Synthetic Street Scenes is a collection of 22 video datasets of street situations that are rare, dangerous or impractical to film: street flooding, snow cover, traffic jams, collisions and near misses, storm damage, blocked bike lanes, tampering with roadside sensors, and civic maintenance problems seen from egocentric and fixed viewpoints. It holds 5,190 clips and 21.7 hours of video. Every clip was generated from a text prompt with MiniMax H3. No frame shows a real place, person or event.

Each dataset pairs its positive classes with matched controls and hard negatives filmed from the same locations, cameras, lighting and weather, so a model cannot separate the classes by scene alone. Labels come from the generation prompts, and every clip carries an automatic audit of whether its label rendered.

<video controls width="100%" src="https://huggingface.co/datasets/mehmetkeremturkcan/SyntheticStreetScenes/resolve/main/assets/dataset_explainer.mp4"></video>

Datasets

All clips are 1344 x 768 pixels, 24 frames per second, 362 frames (15.08 s), H.264 video with a generated AAC soundtrack. Audit pass is the share of clips whose label a vision language model judged visible (details below); it is a lower bound on sets whose evidence is small or brief, such as bike lanes and tampering.

Loading

Each dataset is a configuration with train, validation and test splits. The all configuration concatenates the 22 datasets. Decoding video needs torchcodec:

pip install datasets torchcodec

from datasets import load_dataset

crash = load_dataset("mehmetkeremturkcan/SyntheticStreetScenes", "SynCRASH", split="test")
clip = crash[0]
print(clip["label"], clip["subclass"], clip["event_time_s"])

frames = clip["video"].get_frames_at(indices=[0, 120, 240, 361])   # torchcodec decoder
print(frames.data.shape)                                         # (4, 3, 768, 1344)

Filtering on the audit and on the curated subsets:

train = load_dataset("mehmetkeremturkcan/SyntheticStreetScenes", "all", split="train")
clean = train.filter(lambda r: r["qc_pass"])                  # label judged visible
strict = train.filter(lambda r: r["balanced_subset_qc"])      # same label mix per scene, no cuts, judged visible

The all training split downloads 3,634 clips (about 21 GB). To fetch raw files for one dataset:

huggingface-cli download mehmetkeremturkcan/SyntheticStreetScenes --repo-type dataset \
    --include "data/SynFLOOD/*" --local-dir SyntheticStreetScenes

Structure

data/<Dataset>/<split>/<label>/<id>.mp4
data/<Dataset>/<split>/metadata.parquet     one row per clip
data/<Dataset>/qc_stats.json                audit statistics per label and threshold
data/UnseenClasses/test/...                 evaluation only, see below

Splits are stratified on the label within each dataset, about 70, 15 and 15 percent, with subclasses spread evenly within each label. Every clip has its own prompt. Locations recur across splits, as they would for fixed cameras, so the test splits measure generalization to new events at known places rather than to new places.

Fields

Dataset specific attributes include the collision type, severity and vehicle count (SynCRASH), congestion cause and whether it is in frame (SynJAM), flood type and whether rain is falling (SynFLOOD), snow clearance compliance (SynSNOW), damage type (SynAFTERMATH), obstruction and hazard type (SynBIKELANE), tampering target, method and perpetrator count (SynTIC), the action, manipulation primitive, load and escalation contact of each civic situation, the inspected asset and the standard its grade follows (SynINSPECT), the surface and the reason for its grade (SynTRAVERSE), and camera height and carrier visibility checks for egocentric clips.

Labels

Traffic and street cameras. SynFLOOD grades water depth from 0 (dry, including heavy rain with drains coping) through ponding and kerb to kerb flooding to 3 (above wheel arches, stalled cars). SynSNOW grades road and sidewalk snow from 0 (clear, cold rain, or freshly plowed with kerb banks) to 3 (deep and unplowed); its compliance attribute records whether hydrants, curb ramps and sidewalks were cleared. SynJAM separates traffic states so that onset is learnable: free flow (including dense traffic that keeps moving), jam forming, jammed and clearing. SynCRASH covers 14 collision types at three severities; its near misses involve hard braking and swerves with no contact, and its normal clips include look-alikes such as an ambulance passing or a tow truck at a stalled car. It also holds 40 crashes re-rendered with prompts anchored on the static aftermath (prompt_variant and fix_of in attributes); the 40 clips they replace are kept, and 38 of them fail the audit. SynAFTERMATH grades storm damage from none (including rain only, wind only, and a street after cleanup) to hazard (a whole tree across the road, a crushed car, a flooded underpass). SynBIKELANE labels a lane as blocked (something in it forces riders out), hazard (open but with a door opening, a wrong way rider or a pothole) or clear (including legal loading beside the lane).

Roadside sensors. SynTIC and SynTIC-NYC show a pole mounted traffic camera watching the other corners of its intersection. attack clips show masked individuals covering, spraying, lasing or striking cameras, signal heads and cabinets. hard_negative clips look similar and are benign: utility crews servicing the camera, cold weather balaclavas, birds on the housing, branches striking the mast, masked marches. SynTIC uses seven generic US intersection types; SynTIC-NYC uses composites of Upper Manhattan streetscapes.

Civic maintenance, asset condition and passability. The SynCIVIC family labels a situation by the decision it requires: actionable (a task a maintenance robot could carry out), nominal (nothing to do, including intentional clutter such as a street market or collection day), and out_of_scope (a real problem that needs a person: a downed cable, a burst main, a person asleep in a doorway). The New York City variants add navigation_hazard. The SynINSPECT family grades 20 asset types from 0 (serviceable) to 3 (urgent), anchored to published standards such as the half inch sidewalk trip hazard. The SynTRAVERSE family grades 36 surfaces from 0 (walk normally) through 1 (passable with an adapted gait) to 2 (reroute).

Viewpoints of the civic family. The original sets were prompted as footage from a humanoid robot's head camera, and the generator drew the robot: an automatic check flags robot hands or a robot head at the frame edge in 57 percent of their 1,292 clips. They are kept for research on robot viewpoints. The -EGO sets show the same situations as a plain first person view with no part of the carrier, from 0.6 to 2.1 m; an automatic check flags 10 of their 956 clips, five are confirmed by eye to show a human hand, knee or shoe, and those five are excluded from the curated subsets. The -INFRA sets use fixed municipal cameras. From 2.5 to 5 m, small asset defects do not render, so SynINSPECT-INFRA and SynTRAVERSE-INFRA are previews; the -INFRA-LOW sets repeat them from 1.2 to 1.5 m, which raises the audit pass rate from 47 to 59 percent and from 49 to 50 percent.

How the clips were made

Generation. MiniMax H3 (FL2VA checkpoint, text to video and audio) with the H3 Turbo LoRA at 8 denoising steps, on eight A100 80 GB GPUs. Each prompt combines a header shared by every class of a dataset (camera, lens, mounting height, location, lighting, weather, and the instruction to film one continuous take) with a clause that states what happens. Prompts, seeds and step counts are stored per clip.

Shortcut control. Three label correlated confounds appeared during generation and were removed. Framing instructions that appeared in one class only made the generator cut to close-ups, so cuts predicted the class; all framing text now lives in the shared header, and n_cuts keeps this auditable (37 of 5,190 clips contain a cut). Rejection sampling drifted the per scene label rate; prompt banks are stratified per scene and generation cycles through (scene, label) cells, so any prefix of a run stays balanced. In the civic sets, the camera's approach behaviour was first tied to the class; it is now sampled independently of the label.

Audit. A vision language model (Qwen2.5-VL-7B-Instruct) scored each clip from 0 to 10 on whether the core situation of its label is visible, from four 896 pixel frames centred on event_time_s for events and spread across the clip otherwise. It judges the label, never incidental prompt details, and negatives are judged on the absence of the positive event. Before release, the judge had to score visually verified crash, near miss and attack clips at 6 or above and a known failure at 3 or below; it scored 7, 7, 7 and 0. A clip passes at 6. The judge cannot localize bike lanes or see small scattered debris: hand checks found 14 of 14 SynBIKELANE clear clips and 10 of 10 SynAFTERMATH grade 1 clips correct, against audit rates of 40 and 51 percent. qc_stats.json gives pass rates at thresholds 4 to 8 so the cut can be moved without re-scoring.

Curated subsets. balanced_subset (4,299 clips) keeps an identical label mix in every scene and drops clips with cuts or a confirmed carrier. balanced_subset_qc (1,276 clips) further requires an audit pass.

Held-out classes

The UnseenClasses configuration holds 36 clips of six traffic camera scene types that none of the 22 datasets contains: fog, wildfire smoke, a road construction zone, a sinkhole, a crowd filling the street, and emergency vehicles, six clips each. They were generated with the same pipeline to test how models trained on the collection transfer to situations they never saw. They are not audited and are not part of the 5,190 clips.

Companion model

StreetPrompt is a prompt driven video classifier trained on this collection without class labels. A user types class prompts, for example "the street is flooded" and "a dry street", and the model classifies clips from a street camera. Its smallest variant has 4.35 million parameters, needs 6.0 GMACs per 16 frame clip, and runs on a Jetson Orin Nano.

Zero-shot balanced accuracy on three of its benchmarks; the model card has the full results.

Known limitations

• —Labels are derived from prompts. The generator renders whole frame states such as water depth, snow cover and congestion reliably, and small or brief evidence less reliably: impact damage often leaves equipment intact, and a sheared hydrant or a bent sign can render undamaged. Use qc_pass or the curated subsets where label noise matters.
• —Scenes are composites that evoke real neighbourhoods; none is an identifiable address. Signage text is often garbled.
• —Synthetic footage lacks many real camera artefacts, such as heavy compression, infrared switching and water on the lens. Expect a domain gap and validate on real footage before deployment.
• —Soundtracks are generated with the video and were not audited.

License and attribution

The dataset is released under CC BY 4.0. It may be used, shared and adapted for any purpose, including commercial use, provided that the dataset is credited. Wherever the dataset or material derived from it is used or shared, including papers, model cards, software and products, display the following reference where readers of that work can clearly see it:

Synthetic Street Scenes, Mehmet Kerem Turkcan, Columbia University, Center for Smart Streetscapes (CS3). https://huggingface.co/datasets/mehmetkeremturkcan/SyntheticStreetScenes. Licensed under CC BY 4.0.

For publications, cite the dataset as below.

The clips are outputs of MiniMax H3, used under the MiniMax H3 Community License Agreement; its notice is in NOTICE. MiniMax claims no rights over outputs, but the agreement and its Acceptable Use Policy, including its territorial scope and the requirement to disclose machine generated content, apply to users of the model and its outputs. Every clip is labelled as synthetic in its MP4 metadata.

Citation

@misc{turkcan2026syntheticstreetscenes,
  title        = {Synthetic Street Scenes: Video Datasets of Rare Street Events for Training and Evaluating Video Models},
  author       = {Turkcan, Mehmet Kerem},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/datasets/mehmetkeremturkcan/SyntheticStreetScenes}},
  note         = {Columbia University, Center for Smart Streetscapes (CS3)}
}

Acknowledgements

This work was supported by the NSF Engineering Research Center for Smart Streetscapes under Award EEC-2133516.