Datasetpublic

phonemetransformers/IPA-CHILDES

IPA-CHILDES Dataset This dataset contains utterances downloaded from CHILDES which have been pre-processed and converted to a phonemic representation. Read the paper here. Description Key Columns The scripts used to create the dataset are available here. Many of the columns from CHILDES have been preserved as they are useful for experiments (e.g. number of morphemes, part-of-speech tags, etc.). The key columns added by the processing script are as… See the full description on the dataset page: https://huggingface.co/datasets/phonemetransformers/IPA-CHILDES.

sourceHugging Faceupdated 1y agoView on Hugging Face

7likes295downloads

Dataset Card

IPA-CHILDES Dataset

This dataset contains utterances downloaded from CHILDES which have been pre-processed and converted to a phonemic representation. Read the paper here.

Description

Key Columns

The scripts used to create the dataset are available here. Many of the columns from CHILDES have been preserved as they are useful for experiments (e.g. number of morphemes, part-of-speech tags, etc.). The key columns added by the processing script are as follows:

Column	Description
`processed_gloss`	The pre-processed orthographic utterance. This includes lowercasing, fixing English spelling and adding punctuation marks. This is based on the AOChildes preprocessing.
`ipa_transcription`	A phonemic transcription of the utterance, space-separated with word boundaries marked with the `WORD_BOUNDARY` token.
`character_split_utterance`	A space-separated transcription of the utterance, produced simply by splitting the processed gloss by character. This is intended to have a very similar format to `ipa_transcription` for studies comparing phonetic to orthographic transcriptions.
`is_child`	Whether the utterance was spoken by a child or not. Note that this is set to `False` for all utterances in this dataset, but the processing script has the ability to preserve child utterances.

character_split_utterance and ipa_transcription are designed for training character-based (phoneme-based) language models using a simple tokenizer that splits around whitespace. The processed_gloss column is suitable for word-based (or subword-based) language models with standard tokenizers.

Note that the data has been sorted by the target_child_age column, which stores child age in months. This can be used to limit the training data according to a maximum child age.

Dataset Sections

The following languages are included (ordered by number of phonemes):

Language	Description	Phoible Inventory ID	Speakers	Utterances	Words	Phonemes	% Child
EnglishNA	Taken from 49 corpora in the EnglishNA collection of CHILDES and phonemized using `phonemizer` with language code `en-us`.	2175	3,687	2,564,614	9,993,744	30,986,218	35.83
EnglishUK	Taken from 16 corpora in the EnglishUK collection of CHILDES and phonemized using `phonemizer` with language code `en-gb`.	2252	869	2,043,115	7,147,541	21,589,842	39.00
German	Taken from 10 corpora in the German collection of CHILDES and phonemized using `epitran` with language code `deu-Latn`.	2398	829	1,525,559	5,825,166	21,442,576	43.61
Japanese	Taken from 11 corpora in the Japanese collection of CHILDES and phonemized using `phonemizer` with language code `ja`.	2196	489	998,642	2,970,674	11,985,729	44.20
Indonesian	Taken from 1 corpora in the EastAsian/Indonesian collection of CHILDES and phonemized using `epitran` with language code `ind-Latn`.	1690	438	813,795	2,347,642	9,370,983	34.32
French	Taken from 15 corpora in the French collection of CHILDES and phonemized using `phonemizer` with language code `fr-fr`.	2269	1,277	721,121	2,973,318	8,203,649	40.07
Spanish	Taken from 18 corpora in the Spanish collection of CHILDES and phonemized using `epitran` with language code `spa-Latn`.	164	1,009	533,308	2,183,992	7,742,550	45.93
Mandarin	Taken from 16 corpora in the Chinese/Mandarin collection of CHILDES and phonemized using `pinyin_to_ipa` with language code `mandarin`.	2457	2,118	530,022	2,264,198	6,605,913	38.88
Dutch	Taken from 5 corpora in the DutchAfricaans/Dutch collection of CHILDES and phonemized using `phonemizer` with language code `nl`.	2405	107	403,472	1,475,174	4,786,803	35.08
Polish	Taken from 2 corpora in the Slavic/Polish collection of CHILDES and phonemized using `phonemizer` with language code `pl`.	1046	511	218,860	1,042,841	4,361,797	63.26
Serbian	Taken from 1 corpora in the Slavic/Serbian collection of CHILDES and phonemized using `epitran` with language code `srp-Latn`.	2499	208	319,305	1,052,337	3,841,600	29.14
Estonian	Taken from 9 corpora in the Other/Estonian collection of CHILDES and phonemized using `phonemizer` with language code `et`.	2181	157	186,921	843,189	3,429,228	44.71
Welsh	Taken from 2 corpora in the Celtic/Welsh collection of CHILDES and phonemized using `phonemizer` with language code `cy`.	2406	269	181,292	666,350	1,939,286	69.18
Cantonese	Taken from 2 corpora in the Chinese/Cantonese collection of CHILDES and phonemized using `pingyam` with language code `cantonese`.	2309	95	205,729	777,997	1,864,771	33.54
Swedish	Taken from 3 corpora in the Scandinavian/Swedish collection of CHILDES and phonemized using `phonemizer` with language code `sv`.	1150	41	154,064	581,451	1,782,692	44.63
PortuguesePt	Taken from 4 corpora in the Romance/Portuguese collection of CHILDES and phonemized using `phonemizer` with language code `pt`.	2206	45	134,543	499,522	1,538,408	39.47
Korean	Taken from 3 corpora in the EastAsian/Korean collection of CHILDES and phonemized using `phonemizer` with language code `ko`.	423	127	105,281	263,030	1,345,276	36.76
Italian	Taken from 5 corpora in the Romance/Italian collection of CHILDES and phonemized using `phonemizer` with language code `it`.	1145	109	94,361	352,861	1,309,489	39.02
Croatian	Taken from 1 corpora in the Slavic/Croatian collection of CHILDES and phonemized using `epitran` with language code `hrv-Latn`.	1139	54	90,992	305,112	1,109,696	39.24
Catalan	Taken from 6 corpora in the Romance/Catalan collection of CHILDES and phonemized using `phonemizer` with language code `ca`.	2555	180	89,103	319,726	1,084,594	36.49
Icelandic	Taken from 2 corpora in the Scandinavian/Icelandic collection of CHILDES and phonemized using `phonemizer` with language code `is`.	2568	17	78,181	279,939	1,057,235	35.21
Basque	Taken from 2 corpora in the Other/Basque collection of CHILDES and phonemized using `phonemizer` with language code `eu`.	2161	286	71,537	230,500	942,725	48.82
Hungarian	Taken from 3 corpora in the Other/Hungarian collection of CHILDES and phonemized using `epitran` with language code `hun-Latn`.	2191	116	69,690	237,062	918,002	47.95
Danish	Taken from 1 corpora in the Scandinavian/Danish collection of CHILDES and phonemized using `phonemizer` with language code `da`.	2265	29	84,019	275,170	824,314	41.71
Norwegian	Taken from 2 corpora in the Scandinavian/Norwegian collection of CHILDES and phonemized using `phonemizer` with language code `nb`.	499	34	61,906	227,856	729,649	42.58
PortugueseBr	Taken from 2 corpora in the Romance/Portuguese collection of CHILDES and phonemized using `phonemizer` with language code `pt-br`.	2207	331	22,439	174,845	577,865	44.42
Romanian	Taken from 3 corpora in the Romanian collection of CHILDES and phonemized using `phonemizer` with language code `ro`.	2443	33	54,982	152,465	537,669	42.62
Turkish	Taken from 2 corpora in the Other/Turkish collection of CHILDES and phonemized using `phonemizer` with language code `tr`.	2217	118	29,317	79,404	421,129	50.58
Irish	Taken from 2 corpora in the Celtic/Irish collection of CHILDES and phonemized using `phonemizer` with language code `ga`.	2521	29	27,818	105,867	338,425	34.37
Quechua	Taken from 2 corpora in the Other/Quechua collection of CHILDES and phonemized using `phonemizer` with language code `qu`.	104	14	22,397	46,848	281,478	40.06
Farsi	Taken from 2 corpora in the Other/Farsi collection of CHILDES and phonemized using `phonemizer` with language code `fa-latn`.	516	29	22,613	43,432	178,523	40.45

Papers

This dataset has been used in the following key papers: