Pilin2005/open-customs-data

Open Customs Data

Dataset Summary

This dataset contains 65,163,468 import transactions from the Bureau of Customs of the Philippines spanning from 2012 to 2025. It represents a comprehensive record of goods imported into the Philippines, including detailed information on product classifications, values, duties paid, countries of origin, and ports of entry.

The data was extracted from the Bureau of Customs' electronic2mobile (e2m) system and has been cleaned, standardized, and unified for research purposes. The original 2012-2019 data contains 20,255,360 transactions with 30 variables, while the extended 2020-2025 data adds 44,908,108 transactions. All data has been combined into a unified schema with 37 columns.

Languages

The dataset is primarily in English (en).

Dataset Structure

Data Instances

The unified dataset is available in Parquet format combining all years from 2012 to 2025. The data follows a standardized schema across all years with 37 columns.

Example record structure:

{
  "year": 2012,
  "month": null,
  "quarter": 1,
  "month_code": null,
  "uid": "2012q1 00000001",
  "entry_number": null,
  "hs_code": "15119090000",
  "goods_description": "PALM OLEIN IN BULK",
  "country_origin": "MYS",
  "country_export": "MYS",
  "quantity": 2499867,
  "unit_of_measure": null,
  "unit_price": 0.97359568,
  "gross_mass_kg": null,
  "net_mass_kg": null,
  "currency": "USD",
  "dutiable_value_foreign": 2486856.15,
  "exchange_rate": 43.921,
  "dutiable_value_php": 109225209,
  "customs_value": null,
  "fx_usd": 1,
  "fob_value": 2433859.8,
  "cif_value": 2486856.3,
  "freight": null,
  "insurance": null,
  "arrastre": 274985,
  "wharfage": null,
  "duty": 0,
  "vat_base": 112934962,
  "vat": 13552195,
  "excise_ad_valorem": 0,
  "duties_taxes_total": 13552195,
  "other_tax": null,
  "fines_penalties": null,
  "preference_code": "AFTA",
  "port": null,
  "subport": null
}

Data Fields

The unified schema contains 37 columns standardized across all years (2012-2025):

Data Splits

The complete unified dataset spans 2012-2025 with yearly distribution:

Original Source Data:

• —2012-2019: 8 CSV files totaling 3.8 GB with 20,255,360 transactions
• —2020-2025: 6 yearly partitioned Parquet files totaling 44,908,108 transactions

Parquet Files

The dataset is available in Apache Parquet format for improved query performance and reduced storage.

Unified Combined File (2012-2025)

combined.parquet contains all 65+ million transactions with a unified schema in a single file (4.92 GB).

Best for: Full dataset queries, cross-year analysis, and comprehensive research requiring all historical data.

# Python - Polars
import polars as pl
df = pl.read_parquet('combined.parquet')

# Python - Pandas
import pandas as pd
df = pd.read_parquet('combined.parquet')

# DuckDB
SELECT * FROM 'combined.parquet' WHERE year = 2024 LIMIT 10;

Key Statistics:

• —Total rows: 65,163,468
• —Total columns: 37
• —Unique HS codes: 20,639
• —Date range: 2012-2025
• —Total dutiable value: ₱103 trillion
• —Total duties & taxes: ₱15.3 trillion

Yearly Partitioned Files

yearly/parquet/ contains separate files for each year:

# Read all years
df = pl.read_parquet('yearly/parquet/*.parquet')

# Read specific year
df_2024 = pl.read_parquet('yearly/parquet/2024.parquet')

Dataset Creation

Source Data

Initial Data Collection and Normalization

The data was extracted from the Bureau of Customs' electronic2mobile (e2m) system, which is the official electronic customs processing system in the Philippines.

2012-2019 Data: Original dataset published by Ken Abante (2020), containing 20,255,360 transactions extracted from the e2m system.

2020-2025 Data: Extended dataset obtained from the Bureau of Customs via their OneDrive repository at https://customsph-my.sharepoint.com/personal/delakrusjcustomsgovph/layouts/15/onedrive.aspx?id=%2Fpersonal%2Fdelakrusj%5Fcustoms%5Fgov%5Fph%2FDocuments%2FImport%20Entries&ga=1, containing 44,908,108 additional transactions.

Annotations

Data Enrichment and Unification

To create a unified dataset spanning 2012-2025, the following transformations and annotations were applied:

2020-2025 Data Processing:

• —Year and month extraction: Year and month columns were extracted from the original Excel filenames (format: YYYY-MM) and added as separate columns to enable temporal analysis
• —Source format: Original data provided as Excel (.xlsx) files, one per month, converted to Parquet format
• —Schema standardization: Column names and data types were standardized to match the unified 37-column schema

Schema Unification (2012-2025):

• —Columns from both data sources (2012-2019 and 2020-2025) were mapped to a common schema with 37 unified columns
• —Column name standardization: Original column names were converted to snakecase format (e.g., `dutiablevaluephp` → `dutiablevalue_php`)
• —Data type normalization: All numeric fields standardized to Float64, temporal fields to Int32, and identifiers to String
• —Country code handling: Both ISO 3166-1 alpha-3 codes and full country names are preserved in country_origin and country_export fields
• —Missing column handling: Columns present in one dataset but not the other are filled with null values

Column Mappings:

• —2012-2019: ty → year, tm → month, tq → quarter
• —2012-2019: hscode → hs_code, goodsdescription → goods_description
• —2012-2019: countryorigin_iso3 → country_origin, countryexport_iso3 → country_export
• —2012-2019: p → unit_price, q → quantity
• —2012-2019: dutypaid → duty, vatpaid → vat, vatbase → vat_base
• —2012-2019: m_fob → fob_value, m_cif → cif_value
• —2012-2019: dutiestaxes → duties_taxes_total, prefcode → preference_code
• —2012-2019: finesandpenalties → fines_penalties, othertax → other_tax

Parquet Aggregation

Unified Dataset Generation

The unified Parquet file (combined.parquet) was generated using Polars (>=0.20.0) with the following process:

1. 1.Multi-source ingestion:
2. 2.2012-2019 data: Read from 8 CSV files
3. 3.2020-2025 data: Read from yearly partitioned Parquet files with year/month metadata extracted from source Excel filenames

1. 1.Schema unification: All source data mapped to a common 37-column schema:
2. 2.Numeric fields standardized to Float64 (prices, values, duties, taxes, quantities, masses)
3. 3.Temporal fields standardized to Int32 (year, month, quarter)
4. 4.Text fields standardized to String (codes, descriptions, country identifiers)
5. 5.HS codes preserved as String to maintain leading zeros in tariff codes

1. 1.Column mapping and renaming:
2. 2.Legacy column names (e.g., ty, dutypaid, vatpaid) mapped to unified names (e.g., year, duty, vat)
3. 3.Snake_case naming convention applied consistently across all columns
4. 4.Missing columns in either source filled with null values

1. 1.Data quality handling:
2. 2.UTF-8 lossy encoding applied to handle invalid byte sequences
3. 3.Ragged line handling for inconsistent column counts in CSV sources
4. 4.Null value preservation for missing or incomplete data

1. 1.Streaming aggregation: Lazy evaluation and streaming operations used to process 65+ million rows without exceeding memory constraints

1. 1.Compression: Snappy compression applied to the unified file (4.92 GB total size)

1. 1.Statistics: Column statistics embedded in Parquet metadata to enable query optimization and predicate pushdown

Yearly Parquet Generation

The yearly Parquet files were generated with:

• —Schema normalization ensuring consistent data types across years
• —Snappy compression reducing storage requirements
• —Yearly partitioning for efficient year-specific queries

Additional Information

Citation Information

For the unified dataset (2012-2025):

Bureau of Customs of the Philippines. (2012-2025). Import Transaction Records. Combined and unified dataset.

For the original 2012-2019 dataset:

Abante, Ken. (2020). An Open Data Set of Twenty Million Import Transactions from the Bureau of Customs of the Philippines (2012-2019). doi:10.13140/RG.2.2.24053.73444.

Data Sources:

• —2012-2019: Abante, K. (2020) via ResearchGate
• —2020-2025: Bureau of Customs of the Philippines OneDrive Repository (https://customsph-my.sharepoint.com/personal/delakrusjcustomsgovph/layouts/15/onedrive.aspx?id=%2Fpersonal%2Fdelakrusj%5Fcustoms%5Fgov%5Fph%2FDocuments%2FImport%20Entries&ga=1)