Yiwei-Ou/Urban-ImageNet

🏙️ Urban-ImageNet

A Large-Scale Multi-Modal Dataset and Evaluation Framework for Urban Space Perception from Social Media Imagery.

<p align="center"> <a href="https://arxiv.org/abs/2605.09936"><img src="https://img.shields.io/badge/arXiv-2605.09936-b31b1b.svg" alt="arXiv"/></a> <a href="https://github.com/yiasun/dataset-2"><img src="https://img.shields.io/badge/GitHub-yiasun%2Fdataset--2-black?logo=github" alt="GitHub"/></a> <a href="https://huggingface.co/datasets/yiasun/urban-imagenet"><img src="https://img.shields.io/badge/🤗%20HuggingFace-Dataset-yellow" alt="HuggingFace"/></a> </p>

Urban-ImageNet fills a critical gap between computer vision and urban studies by treating cities not simply as visual scenes, but as lived, socially produced, and experientially activated spaces.

Overview

ImageNet taught models to recognise objects. Urban-ImageNet teaches them to understand how people experience cities.

General-purpose benchmarks such as ImageNet and Places365 identify what is in a scene, but they were never designed to answer the question that matters in urban studies: how do people inhabit, narrate, and socially activate urban space? Urban-ImageNet is a domain-specific complement — a 2-million image–text benchmark drawn from real social media, organised by HUSIC (Hierarchical Urban Space Image Classification), a 10-class taxonomy grounded in the urban theories of Lefebvre, Gehl, and Newman, to capture the spatial, social, and functional distinctions that matter in urban research.

The corpus contains over 2 million public Weibo image–text pairs collected from 61 urban commercial sites across 24 Chinese cities spanning 2019–2025, with controlled benchmark subsets at 1K, 10K, and 100K scale, and a full 2M corpus for large-scale training. The benchmark supports three tasks within one standardised library (Urban-ImageNet-lib):

[image] Figure 1: The Urban-ImageNet framework — addressing current limitations in urban perception evaluation. The dataset bridges general-purpose vision benchmarks and domain-specific urban research needs through the HUSIC taxonomy and three unified benchmark tasks.

Dataset Variants

Four tiers are released to support model development and scaling-behaviour studies:

For all balanced tiers the train/val/test split ratio is 80:10:10. All three tasks share identical image files across tiers; only labels and metadata files differ. The 2M corpus provides per-class image counts to support informed use under realistic class imbalance.

Sample Dataset

A 100-image representative sample of Urban-ImageNet is available as a separate HuggingFace repository for those who wish to quickly inspect data quality, explore file formats, and understand annotation conventions before working with the full corpus.

The sample was constructed by stratified random sampling from the train split of the 100K Dataset: 10 images were drawn uniformly at random from each of the 10 HUSIC classes, yielding a perfectly balanced 100-image subset. All three task annotations are fully included for every sampled image, and instance segmentation mask visualisations are provided for all 100 images.

All 100 images underwent manual human review on top of the automated privacy-protection pipeline. See the Sample Dataset README for full details.

File Structure

Balanced Tiers (1K / 10K / 100K)

{Tier} Dataset/
├── 01 Images with labels/          ← Task 1: Scene Classification
│   ├── train/
│   │   ├── Exterior urban spaces with people/
│   │   │   └── *.jpg
│   │   ├── Exterior urban spaces without people/
│   │   │   └── *.jpg
│   │   ├── Food or drink items/
│   │   ├── Hotel or commercial lodging spaces/
│   │   ├── Human-centered portrait/
│   │   ├── Interior urban spaces with people/
│   │   ├── Interior urban spaces without people/
│   │   ├── Other non-spatial content/
│   │   ├── Private home interiors/
│   │   └── Retail products and merchandise/
│   ├── val/   (same structure)
│   └── test/  (same structure)
│
├── 02 Text-Image Pairs/            ← Task 2: Cross-Modal Retrieval
│   ├── train.xlsx
│   ├── val.xlsx
│   └── test.xlsx
│
└── 03 Instance Segmentation/       ← Task 3: Instance Segmentation
    ├── train.json
    ├── val.json
    ├── test.json

Full Corpus (2M)

Full Dataset-2M/
├── Images/                         ← All 2M+ images (flat, no subfolders)
│   └── *.jpg
└── Labels/
    ├── 01 Semantic classification labels.CSV
    ├── 02 Text-Image Pairs.CSV
    └── 03 Instance Segmentation labels.json

Image format: All released images are JPEG, privacy-protected, and resized to a maximum long edge of 512 px (short edge scaled proportionally). Original usernames, faces, licence plates, and QR codes have been removed or blurred.

The HUSIC Framework

<!-- Replace with actual HUSIC taxonomy figure --> [image] Figure 2: The HUSIC 10-class hierarchical taxonomy. Classes are organised into two primary groups (Spatially Relevant / Non-Spatially Relevant) and five secondary groups. Manual annotation by three trained researchers achieved Cohen's κ = 0.87.

Raw location-tagged social media content is inherently heterogeneous. A user posting under a single hashtag such as #Beijing Sanlitun produces content spanning architectural photography, dining imagery, merchandise displays, selfies, hotel promotion, and noise. Without a principled framework, downstream spatial analyses are confounded by this heterogeneity. HUSIC resolves this by providing a theoretically grounded taxonomy that simultaneously serves as a UGC filtering pipeline and a 10-way classification benchmark.

Theoretical Grounding

HUSIC class boundaries are defined by domain-expert concepts rather than data-driven frequency, drawing on three complementary bodies of urban theory:

• —Lefebvre's Production of Space — The distinction between conceived space (design intent) and lived space (social appropriation through use) motivates the with/without people axis within each spatial group, a distinction absent from all existing vision benchmarks.
• —Gehl's Public Life Studies — Gehl's finding that social activity is both an indicator and a self-reinforcing generator of successful public space justifies treating activated and non-activated spaces as analytically distinct categories.
• —Newman's Spatial Hierarchy — Newman's defensible-space framework, which conceptualises urban environments along a public-to-private gradient, provides the basis for HUSIC's three-tier spatial hierarchy: publicly accessible spaces, transitional semi-public spaces, and privately controlled spaces.

HUSIC Class Definitions

Design Philosophy: A Foundation Framework, Not a Closed Taxonomy

HUSIC intentionally operates at a 10-class foundation level rather than providing exhaustive fine-grained subcategories. This is a deliberate design framework for various downstream urban studies.

The primary purpose of these 10 classes is to serve as a universal filtering and routing layer for social media imagery: they cleanly partition the semantic space of urban UGC so that researchers can isolate the specific subset relevant to their question, discard noise, and then apply domain-specific analysis on a purified corpus. The downstream subcategorisation — which varies enormously across research questions — is intentionally left open for the research community to define.

Filtering to a single HUSIC class typically eliminates up to 90% of irrelevant content before any domain-specific analysis begins, dramatically improving the signal-to-noise ratio regardless of the downstream task. To further support deeper analysis within each class, HUSIC is complemented by Task 3 instance segmentation, which provides 12–20 class-specific object labels per HUSIC category. This gives researchers both a clean high-level routing layer and a set of object-level semantic anchors without requiring exhaustive fine-grained annotation at the dataset level.

Some examples of how individual HUSIC classes can seed specialised downstream research:

Rationale: Spatially Relevant Classes (IDs 0–5)

The spatially relevant classes cover the full spectrum of urban environments from public exterior, through public interior, to semi-public and private spaces. The with/without people split within exterior and interior classes is analytically essential:

• —With people classes (IDs 0, 2) enable research on pedestrian behaviour, social activity patterns, human action recognition, and spatial vitality measurement. With pose estimation and action recognition techniques, these images can reveal how different user groups occupy public space and what kinds of activities different spatial configurations support.
• —Without people classes (IDs 1, 3) isolate architectural and design elements independent of human activity — supporting aesthetic perception studies, visual quality assessment, and design feature analysis without the confound of human occlusion.

The accommodation classes (IDs 4–5) capture a phenomenon consistently observed in the corpus: users post hotel and private-home interiors alongside commercial-district imagery, reflecting how urban commercial centres influence both short-term (hotel / Airbnb) and long-term (residential) rental markets in their surroundings — a research direction not addressed by any existing benchmark.

Rationale: Non-Spatially Relevant Classes (IDs 6–9)

Non-spatial classes capture the consumption and social dimensions of urban life equally present in social media posts about commercial districts, even when images contain no spatial content.

• —ID 6 — Food or drink items: UGC food images carry implicit preference signals — frequency of posting reflects popularity and memorability without requiring explicit ratings. Useful for F&B market research and restaurant analytics.
• —ID 7 — Retail products and merchandise: Product images reveal consumer preferences and brand visibility. Because posts originate from people who visited specific commercial sites, the corpus carries an implicit spatial anchor useful for retail market analysis.
• —ID 8 — Human-centered portrait: Self-portraits and group photos document the social occasions that spaces enable — revealing which settings activate gathering, self-expression, and place attachment. Analysing when and where people choose to photograph themselves is itself evidence of spatial vitality.
• —ID 9 — Other non-spatial content: A residual noise class — advertisements, screenshots, memes, infographics. Minimal relevance for urban or commercial research; its isolation as a discrete class makes downstream filtering reliable and auditable.

Task 1: Urban Scene Semantic Classification

Goal: Given an input image, predict its HUSIC label (class ID 0–9).

File format: ImageFolder-style hierarchy under 01 Images with labels/. The subdirectory name is the ground-truth label. Integer labels 0–9 follow lexicographic sort, directly compatible with PyTorch torchvision.datasets.ImageFolder.

from torchvision.datasets import ImageFolder
from torchvision import transforms

dataset = ImageFolder(
    root="100K Dataset/01 Images with labels/train",
    transform=transforms.Compose([
        transforms.Resize(224),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
    ])
)
# dataset.classes → ['Exterior urban spaces with people', 'Exterior urban spaces without people', ...]
# dataset.class_to_idx → {'Exterior urban spaces with people': 0, ...}

T1 Baseline Results (100K benchmark, 80K/10K/10K split)

CLIP zero-shot performs poorly because HUSIC labels such as activated exterior space and non-activated interior space are not standard web image categories. Fine-tuning substantially improves CLIP, but it remains below supervised classifiers. The interior-without-people vs. interior-with-people boundary is the most challenging distinction across all models.

Task 2: Cross-Modal Image–Text Retrieval

Goal: Given a text query, retrieve matching images (text-to-image), or given an image, retrieve matching text (image-to-text).

File format: Three Excel spreadsheets (train.xlsx, val.xlsx, test.xlsx) under 02 Text-Image Pairs/. Each row describes one image and its associated Weibo post metadata.

Metadata Schema

To maximise privacy protection and minimise re-identification risk, the public release contains only three essential columns:

All other fields, including encrypted User IDs, Post IDs, timestamps, City Name, Place Tag, Posting Tool, Mentioned Users, Extracted Topics/Locations, and all engagement metrics (likes, reposts, comments) — have been completely removed from the public release to ensure maximum privacy protection. The remaining three columns provide the minimum information necessary to support the three benchmark tasks while making re-identification practically impossible.

Note: The released dataset retains original Chinese text in the Post Text column to preserve linguistic authenticity and avoid translation distortion, which is scientifically important for Task 2 evaluation. English text in this README is for illustrative purposes only.

Two Retrieval Sub-Tasks

Urban-ImageNet supports two complementary retrieval configurations that reflect increasing real-world difficulty:

Bidirectional use: Both sub-tasks support either direction:

• —Image → Text: given an image, retrieve its HUSIC label or matching post text.
• —Text → Image: given a HUSIC label or post text, retrieve the matching image(s). For T2-B, one post may correspond to up to 9 images, so evaluation must use a multi-positive retrieval protocol rather than assuming one-to-one caption–image correspondence.

T2 Baseline Results (10K test set)

Category-label retrieval is near-trivial after fine-tuning (≥92% R@1), confirming that HUSIC descriptions provide strong cross-modal signal. Post-text retrieval is substantially harder: Weibo posts are short informal narratives (median 32 characters) rather than image descriptions, and a single post may accompany images spanning multiple HUSIC classes. Against a random-chance baseline of ~0.1% R@1, fine-tuned CLIP achieves 8.1% (76× chance), establishing a concrete baseline for future urban-domain vision–language models.

[image] Figure 3: T2 retrieval results (avg. T2I + I2T). Category-label retrieval (left) is near-trivial after fine-tuning; post-text retrieval (right) remains genuinely challenging, establishing an important open problem for urban-domain vision–language research.

Task 3: Instance Segmentation

Goal: Detect and delineate urban-domain objects within each image using pixel-level instance masks.

File format: Three COCO-compatible JSON files (train.json, val.json, test.json) under 03 Instance Segmentation/.

Annotation JSON Structure

Each JSON file follows the COCO format with the following fields:

{
  "info": {
    "description": "Urban-ImageNet Instance Segmentation Annotations",
    "split": "train",
    "version": "1.0"
  },
  "categories": [ {"id": 0, "name": "Exterior urban spaces with people"}, ... ],
  "images": [
    {
      "id": 0,
      "file_name": "2668383_2020-01-21_0.jpg",
      "width": 512,
      "height": 384,
      "classification_label": 0
    }, ...
  ],
  "annotations": [
    {
      "id": 0,
      "image_id": 0,
      "category_id": 0,
      "detected_label": "person",
      "detection_score": 0.8732,
      "bbox": [x, y, width, height],
      "area": 4512,
      "segmentation": { "counts": "...", "size": [384, 512] },
      "iscrowd": 0
    }, ...
  ]
}

Extended fields beyond standard COCO:

• —classification_label (in images): the HUSIC class ID of the image — enables multi-task joint training and evaluation.
• —detected_label (in annotations): the specific object term detected by Grounding DINO (e.g., "person", "retail shelf", "escalator").
• —detection_score (in annotations): Grounding DINO confidence score, enabling downstream threshold-based filtering.
• —Segmentation masks are stored in COCO RLE format (run-length encoding), directly compatible with pycocotools.

Annotation Pipeline

Annotations were generated using a two-stage automatic pipeline followed by human quality control:

1. 1.Grounding DINO (text-prompted open-vocabulary object detection) identifies bounding boxes using class-specific vocabulary prompts.

1. 1.SAM 2 (Segment Anything Model 2) refines each detected box into a pixel-level instance mask.

1. 1.NMS (Non-Maximum Suppression) removes overlapping detections.

1. 1.Area filtering removes very small (noise) and very large (full-image) detections.

1. 1.Human review was applied to the evaluation subset, with stricter confidence thresholds (≥0.50 detection score, ≥0.88 IoU), ensuring reliable ground truth for model comparison.

Training pseudo-labels use more permissive thresholds (≥0.35, ≥0.80). Users should account for the pseudo-label nature of annotations when interpreting segmentation performance.

Per-Class Segmentation Vocabulary

Each HUSIC class uses a tailored vocabulary of 12–20 object terms designed to capture the semantically appropriate instances for that scene type, maximising detection recall while minimising false positives.

T3 scope note: Instance segmentation masks are generated for all 10 HUSIC classes. The T3 evaluation benchmark adopts a class-agnostic protocol — treating all detected objects as a single object category — to produce conservative, architecture-comparable metrics uncorrupted by class-imbalanced pseudo-labels. Per-class AP results are available in the supplementary material of the paper.

T3 Baseline Results (quality-filtered evaluation subset, confidence ≥ 0.50, IoU ≥ 0.88)

†GT-box SAM uses ground-truth bounding boxes as prompts — an oracle upper bound, not a trainable baseline. Adding SAM box-refinement to Mask R-CNN increases AP by ~40% relative (0.267 → 0.373), establishing a strong open-source baseline for future work. The gap between trainable models and the oracle (0.373 vs. 0.749) highlights substantial room for improvement in urban commercial-space instance segmentation.

<!-- Replace with actual T3 qualitative figure --> [image] Figure 4: Task 3 qualitative segmentation examples across HUSIC classes. Colour-coded instance masks from Mask R-CNN, Cascade Mask R-CNN, and Mask R-CNN+SAM. The domain-specific vocabulary enables detection of urban-specific objects (escalators, retail shelves, display cases, street lamps) not well-covered by general segmentation benchmarks.

Urban-ImageNet-lib

<!-- Replace with actual lib architecture figure --> [image] Figure 5: Urban-ImageNet-lib architecture — a unified benchmarking framework supporting all three tasks with standardised cross-dataset comparison adapters.

Urban-ImageNet-lib is a Python benchmarking library providing:

• —Modular data loaders for all three tasks and all four dataset tiers.
• —Standard fine-tuning pipelines for T1 (classification), T2 (retrieval), and T3 (segmentation) baselines.
• —Evaluation scripts with metrics matching established benchmarks (T1 ↔ Places365/SUN; T2 ↔ MS-COCO Captions/Flickr30K; T3 ↔ MS-COCO Instance Seg./Cityscapes).
• —Cross-dataset adapters enabling direct performance comparison in a unified table.

See the GitHub repository for full installation instructions and usage examples.

Scaling Behaviour

Urban-ImageNet's four-tier design enables systematic study of how classification accuracy and computational cost scale with dataset size. All balanced tiers (1K / 10K / 100K) are strictly class-balanced so that performance differences across tiers are attributable to data quantity alone, without confounding from class imbalance. All models were trained separately on each tier and evaluated on a shared held-out 10K test set.

T1 Scaling: Top-1 Accuracy and Macro-F1

† LLaVA-1.5 100K fine-tuning was not completed due to computational constraints (~3,200× slower per sample than ResNet-50; estimated >150 GPU-hours on H100).

All models improve monotonically with scale. The 1K→10K gain (10–12%) consistently exceeds the 10K→100K gain (5%), consistent with standard scaling laws. LLaVA-1.5's stronger language-grounded priors give it an advantage at small scales (76.8% at 1K vs. 66.5–70.8% for others) but it is computationally prohibitive at 100K.

Hierarchical T1 Scaling: Coarser Distinctions Are Easier

HUSIC's hierarchical structure means models can be evaluated at three levels of granularity. At 100K, models substantially exceed their 10-class accuracy when evaluated on coarser distinctions:

At 100K, spatial vs. non-spatial binary accuracy reaches 94% and exterior vs. interior reaches 95%, confirming that HUSIC captures semantically meaningful hierarchical structure. The gap between coarse (94–95%) and fine-grained (83–85%) accuracy highlights that the activation-level distinctions (e.g., with people vs. without people) remain the hardest sub-problems.

T2-Post Retrieval Scaling

Post-level retrieval difficulty grows naturally as the candidate gallery expands. Fine-tuned CLIP's average R@1 drops from 39.5% on the 1K split (100-image pool) to 8.1% on the 10K split (1,000-image pool), confirming that T2-B is a scalably challenging benchmark.

(Avg. = average of T2I and I2T directions; mAP as a fraction 0–1.)

Data Collection and Construction Pipeline

<!-- Replace with actual pipeline figure --> [image] Figure 6: Overview of the Urban-ImageNet dataset construction and annotation pipeline — from Weibo crawling through privacy processing, HUSIC annotation, and multi-task organisation.

Urban-ImageNet was constructed through a five-stage pipeline:

1. 1.Collection — A Python-based web crawler systematically retrieved all public Weibo posts from location-specific hashtags at 61 major urban commercial sites across 24 Chinese cities, covering 2019–2025. Up to 9 image attachments, post text, and metadata were captured per post, yielding a raw corpus of over 4 TB and 2 million image–text pairs.

1. 1.Cleaning — Four-stage deduplication and filtering: (i) near-duplicate removal via perceptual hashing (pHash, Hamming distance ≤ 8); (ii) discard of images smaller than 256×256 px; (iii) NSFW filtering via pre-trained classifier; (iv) removal of systematically repeated commercial advertisement posts via post-text hash similarity.

1. 1.Privacy Protection — Automated face detection, licence-plate recognition, and QR-code detection were applied to all images with all detected regions blurred. Original usernames were stripped and replaced with opaque numerical identifiers. Images were resized to a maximum side length of 512 px. The raw 4 TB corpus is retained securely by the authors and will not be publicly released.

1. 1.HUSIC Annotation (T1 & T2) — The 100K balanced benchmark set was manually annotated by three trained researchers following a standardised guideline. A shared 3,000-image double-annotation subset yielded Cohen's κ = 0.87 (near-perfect agreement). Disagreements were resolved by majority vote and guideline revision. The annotation process took approximately two years of sustained effort.

1. 1.Instance Segmentation (T3) — Pseudo-labels were generated using Grounding DINO + SAM 2 with per-class vocabulary prompts, followed by NMS and area filtering. The evaluation subset was reviewed with stricter thresholds and human spot-checks.

Geographic and Site Coverage

Urban-ImageNet covers 61 urban commercial sites across 24 Chinese cities spanning 8 macro-regions, including all four first-tier cities (Beijing, Shanghai, Guangzhou, Shenzhen), leading new first-tier cities (Chengdu, Hangzhou, Nanjing, Wuhan, Xi'an, Chongqing), and a range of second-tier regional centres. Sites span three spatial typologies: enclosed malls, open-air pedestrian precincts, and mixed-typology developments. The full site list is provided in the paper appendix.

<!-- Replace with your actual geographic distribution figure --> [image] Figure 7: Geographic distribution of Urban-ImageNet's 24 collection cities. Marker size is proportional to the number of collected image–text pairs per city; colour encodes macro-region.

Privacy and Responsible Use

Urban-ImageNet is derived from public Weibo posts — posts whose visibility was explicitly set to "open to all" by the account holder at the time of collection. Although source posts were public, the released dataset applies multiple layers of privacy protection in line with the practice of large-scale street-level datasets (e.g., Google Street View):

Data-use agreement: Researchers accessing Urban-ImageNet must agree to a data-use agreement restricting use to non-commercial academic research and prohibiting:

• —Re-identification of individuals

• —Facial recognition or biometric profiling

• —Account or identity reconstruction

• —Surveillance or social scoring

• —Law-enforcement targeting

• —Commercial profiling or demographic inference

Research purpose: Urban-ImageNet is designed to advance evidence-based urban design and planning through improved AI perception of public spaces — serving a clear public good. The authors will monitor dataset use and reserve the right to retract access in cases of misuse.

Limitations and Known Biases

• —Geographic bias: The corpus is entirely China-sourced and should not be treated as globally representative of urban commercial spaces.
• —Platform bias: Weibo users are not representative of all city residents; the dataset over-represents younger, urban, mobile-connected demographics.
• —Visual selection bias: Social media images over-represent photogenic, popular, and personally meaningful scenes; empty or mundane spaces are systematically underrepresented.
• —Linguistic bias: Post text is original Chinese social-media language containing slang, emoji, hashtags, and frequently loose image–text coupling.
• —Class imbalance in 2M corpus: The full corpus reflects natural posting frequencies and is significantly class-imbalanced; the balanced 1K/10K/100K tiers do not reflect natural class distributions.
• —T3 pseudo-labels: Task 3 annotations are model-generated pseudo-labels (Grounding DINO + SAM 2), not exhaustive human pixel-level labels; users should account for this when training or evaluating segmentation models.
• —Temporal scope: Posts span 2019–2025; urban commercial environments evolve over time and some sites may have changed significantly.

Related Work

Urban-ImageNet is designed as a domain-specific complement to the following general-purpose benchmarks:

Citation

If you use Urban-ImageNet in your research, please cite our paper:

@article{ou2026urbanimagenet,
  title   = {Urban-ImageNet: A Large-Scale Multi-Modal Dataset and Evaluation Framework for Urban Space Perception},
  author  = {Ou, Yiwei and Cheung, Chung Ching and Ang, Jun Yang and Ren, Xiaobin and Sun, Ronggui and Gao, Guansong and Zhao, Kaiqi and Manfredini, Manfredo},
  journal = {arXiv preprint arXiv:2605.09936},
  year    = {2026},
  eprint  = {2605.09936},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url     = {https://arxiv.org/abs/2605.09936}
}

Paper: arXiv:2605.09936 Dataset: huggingface.co/datasets/Yiwei-Ou/Urban-ImageNet Benchmark code: github.com/yiasun/dataset-2

License

The dataset is released under Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0).

You are free to use, share, and adapt this dataset for non-commercial academic research with appropriate attribution, provided that you give appropriate credit and distribute any derivative works under the same license. Commercial use of any kind is prohibited.

See LICENSE for full terms.