electricsheepasia/asia-demographics-typhoon-mangkhut-2018-twitter-data

Typhoon Mangkhut 2018 Twitter Data

Publisher: Qatar Computing Research Institute · Source: HDX · License: cc-by · Updated: 2024-09-13

Abstract

This is a Twitter dataset collected during the typhoon Mangkhut 2018 in the Philippines. The data was collected, processed, and analyzed by the AIDR (http://aidr.qcri.org) platform using state of the art machine learning techniques. The data includes the reports of number of injured and dead people, infrastructure damage reports, missing or found people, urgent needs and donation offers for each hour. Due to Twitter TOS, we do not share full tweets content on HDX. Please contact us via HDX or on aidr.qcri@gmail.com to get tweet ids of the dataset along with a tool which can be used to rehydrate tweets from tweet ids.

Each row in this dataset represents time-series observations. Temporal coverage is indicated by the date column(s). Geographic scope: CHN, GUM, PHL.

Curated into ML-ready Parquet format by [Electric Sheep Africa](https://huggingface.co/electricsheepafrica).

Dataset Characteristics

Variables

Geographic — sympathy_and_support_tweets (range 1.0–1640.0).

Temporal — date.

Demographic — affected_individual_tweets (range 0.0–581.0), infrastructure_and_utilities_damage_tweets (range 0.0–1069.0), number_of_images (range 0.0–133.0), mild_damage_images (range 0.0–38.0), severe_damage_images (range 0.0–24.0).

Outcome / Measurement — number_of_tweets (range 12.0–5518.0).

Identifier / Metadata — esa_source (HDX), esa_processed (2026-05-06).

Other — hour (range 0.0–23.0), injured_or_dead_people_tweets (range 0.0–164.0), missing_and_found_people_tweets (range 0.0–1.0), caution_and_advice_tweets (range 0.0–1104.0), donation_and_volunteering_tweets (range 0.0–113.0) and 1 others.

Quick Start

from datasets import load_dataset

ds    = load_dataset("electricsheepafrica/asia-demographics-typhoon-mangkhut-2018-twitter-data")
train = ds["train"].to_pandas()
test  = ds["test"].to_pandas()

print(train.shape)
train.head()

Schema

Numeric Summary

Curation

Raw data was downloaded from HDX via the CKAN API and converted to Parquet. Column names were lowercased and standardised to snake_case. Common missing-value markers (N/A, null, none, -, unknown, no data, #N/A) were unified to NaN. 14 column(s) were cast from string to numeric or datetime based on parse-success rate (>85% threshold). The dataset was split 80/20 into train and test partitions using a fixed random seed (42) and saved as Snappy-compressed Parquet.

Limitations

• —Data originates from Qatar Computing Research Institute and has not been independently validated by ESA.
• —Automated cleaning cannot correct for misreported values, definitional inconsistencies, or sampling bias in the original collection.
• —This dataset spans 3 countries; geographic and methodological inconsistencies across national boundaries may affect cross-country comparability.
• —Refer to the original HDX dataset page for the publisher's own methodology notes and caveats.

Citation

@dataset{hdx_asia_demographics_typhoon_mangkhut_2018_twitter_data,
  title     = {Typhoon Mangkhut 2018 Twitter Data},
  author    = {Qatar Computing Research Institute},
  year      = {2024},
  url       = {https://data.humdata.org/dataset/typhoon-mangkhut-2018-twitter-data},
  note      = {Repackaged for machine learning by Electric Sheep Africa (https://huggingface.co/electricsheepafrica)}
}

[Electric Sheep Africa](https://huggingface.co/electricsheepafrica) — Africa's ML dataset infrastructure. Lagos, Nigeria.