lang_detect
openwebtext2-first-30-chunks-lang-detect-raw-output
Counting bilingual and monolingual instances
In order to count bilingual and monolingual instances, we use the following code. We count bilingual instances where there are two languages, one of them is English and the other is either German, French, Spanish, Italian, Portuguese or Dutch. All other instances fall into the "Other" category.
from datasets import load_dataset
import json
from tqdm import tqdm
#Specify the dataset name
dataset_name =… See the full description on the dataset page: https://huggingface.co/datasets/RaiBP/openwebtext2-first-30-chunks-lang-detect-raw-output.conv-lang-detect
conv-lang-detect
A multilingual dataset for conversational turn-level language detection — detecting which language a user prefers in a given turn, even when the turn contains pasted content (code, emails, articles, lyrics) in a different language.
The Problem
Standard language detectors return the dominant language of the full text. In a conversational setting, users often paste foreign content and ask a question in their own language:
User (zh-Hant):… See the full description on the dataset page: https://huggingface.co/datasets/ZoneTwelve/conv-lang-detect.new_lang_detection_datasetRaiBP-openwebtext2-first-30-chunks-lang-detect-raw-outputCe répertoire est vide, il a été créé pour améliorer le référencement du jeu de données https://huggingface.co/datasets/RaiBP/openwebtext2-first-30-chunks-lang-detect-raw-output.
holipet-lang-detect-model