datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
GeoGPT_Training_Data_from_Geoscience_Subset_of_CommonCrawl
Description
This dataset is a geoscience-specific subset of CommonCrawl used for GeoGPT training. CommonCrawl is a free and open repository of web crawl data with over 250 billion web pages and is widely used by leading large language models. We apply data mining algorithms to extract geoscience-related content from this vast dataset.
This dataset comprises 12,414,268 samples, each containing the following metadata to trace the data source within CommonCrawl:
id (string):… See the full description on the dataset page: https://huggingface.co/datasets/GeoGPT-Research-Project/GeoGPT_Training_Data_from_Geoscience_Subset_of_CommonCrawl.GeoGPT-QA-RU
GeoGPT-QA-RU — Геологический QA датасет на русском языке
Русскоязычная версия датасета GeoGPT-QA для дообучения LLM в области геологии и нефтегазовой отрасли.
Описание
41,432 пар вопрос-ответ по геонаукам
81.7% переведены на русский язык, 18.3% остались на английском (fallback)
Формат: chat messages (system/user/assistant) — готов для SFT
Перевод выполнен с помощью Gemma-2-27B-IT
Формат данных
JSONL, каждая строка:
{
"messages": [
{"role": "system"… See the full description on the dataset page: https://huggingface.co/datasets/RISEF/GeoGPT-QA-RU.geogpt-chatml-ru5k-dataset-geogpt-fineweb5k-dataset-geogpt-fineweb-random