Datasetpublic

GeoGPT-Research-Project/GeoGPT_Training_Data_from_Geoscience_Subset_of_CommonCrawl

Description This dataset is a geoscience-specific subset of CommonCrawl used for GeoGPT training. CommonCrawl is a free and open repository of web crawl data with over 250 billion web pages and is widely used by leading large language models. We apply data mining algorithms to extract geoscience-related content from this vast dataset. This dataset comprises 12,414,268 samples, each containing the following metadata to trace the data source within CommonCrawl:… See the full description on the dataset page: https://huggingface.co/datasets/GeoGPT-Research-Project/GeoGPT_Training_Data_from_Geoscience_Subset_of_CommonCrawl.

sourceHugging Faceupdated 1y agoView on Hugging Face

1likes38downloads

settings

This repository belongs to GeoGPT-Research-Project on Hugging Face.

CoolFace never edits a repository it does not host. Visibility, licence, collaborators and gating are all managed at the source.

nameGeoGPT_Training_Data_from_Geoscience_Subset_of_CommonCrawl

visibilitypublic

licencenot set

gatedno

ownerGeoGPT-Research-Project

Account settings