openwebtext
RaiBP_-_gpt2-openwebtext2-first-30-chunks-ablation-bilingual-ggufRaiBP_-_gpt2-openwebtext2-first-30-chunks-ablation-full-ggufjax-with-mha-bias-openwebtextcse8803-hw1-openwebtext-gqasemsimula-fock-parflm-depthcond-vtheta-openwebtextGrogros-dmWM-LLama-3-1B-Harm-ft-HarmData-AlpacaGPT4-OpenWebText-d4-a0.25-ft-learnability_advllama_3.2_1b_openwebtext_2025_03_02_converted_fne_gsm8k_2025_03_11dmWM-meta-llama-Llama-3.2-1B-Instruct-ft-HarmData-AlpacaGPT4-OpenWebText-RefusalData-d4-a0.25
openwebtext
Dataset Card for "openwebtext"
Dataset Summary
An open-source replication of the WebText dataset from OpenAI, that was used to train GPT-2.
This distribution was created by Aaron Gokaslan and Vanya Cohen of Brown University.
Supported Tasks and Leaderboards
More Information Needed
Languages
More Information Needed
Dataset Structure
Data Instances
plain_text
Size of downloaded dataset… See the full description on the dataset page: https://huggingface.co/datasets/Skylion007/openwebtext.openwebtext-10kAn open-source replication of the WebText dataset from OpenAI.
This is a small subset representing the first 10K records from the original dataset - created for testing.
The full 8M-record dataset is at https://huggingface.co/datasets/openwebtextopenwebtext-t5Skylion007-openwebtext-tokenizer-gpt2openwebtext-100k
Dataset Card for "openwebtext-100k"
More Information needed
openwebtext2-first-30-chunks-lang-detect-raw-output
Counting bilingual and monolingual instances
In order to count bilingual and monolingual instances, we use the following code. We count bilingual instances where there are two languages, one of them is English and the other is either German, French, Spanish, Italian, Portuguese or Dutch. All other instances fall into the "Other" category.
from datasets import load_dataset
import json
from tqdm import tqdm
#Specify the dataset name
dataset_name =… See the full description on the dataset page: https://huggingface.co/datasets/RaiBP/openwebtext2-first-30-chunks-lang-detect-raw-output.