Datasetpublic

Zihao-Li/IEA_Energy_Dataset

IEA_Energy_Dataset Dataset Details Dataset Description The dataset is energy-related, covering topics of Oil, Coal, Wind, Hydrogen, Bioenergy, Electric vehicles, Heating, Building envelopes, Methane abatement and Chemicals. Dataset Creation Source Data The dataset sources are reports from the webiste of International Energy Agency(IEA). Data Collection and Processing We scraped free open reports from… See the full description on the dataset page: https://huggingface.co/datasets/Zihao-Li/IEA_Energy_Dataset.

sourceHugging Facemitupdated 2y agoView on Hugging Face

5likes296downloads

Dataset Card

IEAEnergyDataset

Dataset Details

Dataset Description

The dataset is energy-related, covering topics of Oil, Coal, Wind, Hydrogen, Bioenergy, Electric vehicles, Heating, Building envelopes, Methane abatement and Chemicals.

Dataset Creation

Source Data

The dataset sources are reports from the webiste of International Energy Agency(IEA).

Data Collection and Processing

We scraped free open reports from IEA's website. The reports are all pdf files and then we used the Llama 3 model to extract useful texts.\ After getting the raw text, we have the two following steps:

1.Remove too short sentences(e.g length < 100)
2.Use Monocleaner to detect disfluent sentences. Each sentence will have a score(0~1), and then we set a threshold to filter.