Datasetpublic

NagaYu/tessera-mixed-pages

Tessera mixed-page corpus Synthetic multilingual web pages with exact span boundaries and multi-label language annotations, for training and evaluating span-level language identification. Why this is synthetic, stated up front No public corpus annotates span boundaries and languages inside real multilingual web pages at scale. Annotating one by hand across 479 languages, most of them low-resource, is not feasible and would itself be error-prone in exactly the… See the full description on the dataset page: https://huggingface.co/datasets/NagaYu/tessera-mixed-pages.

sourceHugging Facecc-by-sa-4.0updated 19d agoView on Hugging Face

0likes46downloads

Dataset Card

Tessera mixed-page corpus

Synthetic multilingual web pages with exact span boundaries and multi-label language annotations, for training and evaluating span-level language identification.

Why this is synthetic, stated up front

No public corpus annotates span boundaries and languages inside real multilingual web pages at scale. Annotating one by hand across 479 languages, most of them low-resource, is not feasible and would itself be error-prone in exactly the places that matter most.

So the gold labels here are constructive: each document is assembled from source sentences of known language, and each span's offsets and language are recorded as it is placed. The boundaries are exact by definition.

The cost is real and is not hidden: the difficulty of this corpus was chosen by its authors. Results measured on it are labelled synthetic wherever they appear, and the accompanying repository reports real-data checks separately.

split	documents	spans	languages	spans <= 20 chars	multi-label spans
train	30,000	561,368	578	52.89%	16.47%
validation	3,000	56,117	534	55.08%	18.25%
test	4,000	74,200	549	55.08%	18.31%
test_adversarial	2,500	47,086	576	55.62%	25.49%
test_romanized	2,500	47,162	114	48.58%	8.66%

test_adversarial applies every degradation layer aggressively. test_romanized contains only non-Latin-script body languages, romanised with probability 1.0.

How a document is built

Pages imitate real furniture, because that furniture is why document-level LID discards low-resource text: navigation, footers, advertising and copyright lines are very often English even on a page whose body is not.

Archetypes:

—buried_body -- body in a mid/tail language, chrome in English. Document-level LID labels the page English, and the body is lost.
—buried_minority -- body in English with a quote or comment block in another language. The document survives filtering as English, and the minority text is silently absorbed into an English corpus.
—codeswitch_heavy, short_span_heavy, monolingual (control).

Degradation layers

shorten (5-20 chars), informalize (repetition, emoji, case, punctuation, abbreviation, typos -- all script-generic, no English-specific slang), romanize, codeswitch, mojibake (encoding round-trip damage), html_residue. Each span records which layers were applied, so results can be stratified by condition.

Multi-label definition

Two kinds of multi-label, scored differently, and which labels are correct is measured rather than asserted:

—`any` (alternatives). A string attested in several languages' source corpora gets all of them. The token Massa is attested in dozens of Latin-script languages; any one is correct, and predicting all of them earns no extra credit.
—`all` (code-switching). The span really contains every listed language, so each is separately required.
—`script_only`. Digits, punctuation and symbols get und and are scored on whether a system declines to name a language.

primary always records the language the text was actually sampled from, so recall-per-language can never be inflated by permissive alternates.

Sources and licensing

source	licence	redistributable	share-alike
UDHR-LID (cis-lmu/udhr-lid)	CC0-1.0	True	False
FLORES-200	CC-BY-SA-4.0	True	True

Effective licence: CC-BY-SA-4.0. FLORES-200 is CC-BY-SA-4.0 and its share-alike term propagates to any corpus containing its sentences.

No scraped web text is redistributed. Only material whose licence permits redistribution is published.

Resource tiers

Defined operationally, not editorially: head = in fastText lid.176's label space, mid = absent from that but present in GlotLID or OpenLID, tail = covered by no evaluated document-level LID. Counts: {'tail': 50, 'mid': 282, 'head': 147}.

Known weaknesses

Please read these before using this corpus for anything consequential.

1.Register. UDHR is formal legal prose and FLORES is encyclopaedic. Neither is conversational. The informalize layer approximates informality; it does not sample it from real usage.
2.Romanisation is systematic transliteration, not attested user romanisation. Real users writing Hindi or Tigrinya in Latin script do not follow a transliteration standard. Anything trained or evaluated on this layer is measuring an approximation, and Han, Kana and Hangul are excluded entirely because character-reading transliteration produces text no human would write.
3.Page structure is templated. Five archetypes cannot cover the real diversity of the web.
4.Sentence counts are very uneven. Some languages have 40 UDHR sentences and others have 997 FLORES sentences; the per-language table below gives the exact figure, and any per-language claim should be read against it.
5.Language identity is contested for some codes. Where a macrolanguage and its varieties both appear, the labels follow the source corpus's choices, which are not neutral and may not match a speaker community's own view.
6.Tail languages are only 50 of the 479 here. GlotLID covers 2,102 labels, so the population most at risk in this corpus is mid: languages that existing LID knows but that document-level application discards anyway.

Language coverage

code	script	source sentences	sources
`aar_Latn`	Latn	57	udhr
`abk_Cyrl`	Cyrl	57	udhr
`ace_Arab`	Arab	997	flores
`ace_Latn`	Latn	1058	flores, udhr
`acm_Arab`	Arab	997	flores
`acq_Arab`	Arab	997	flores
`acu_Latn`	Latn	180	udhr
`ada_Latn`	Latn	59	udhr
`ady_Cyrl`	Cyrl	56	udhr
`aeb_Arab`	Arab	997	flores
`afr_Latn`	Latn	1057	flores, udhr
`agr_Latn`	Latn	57	udhr
`aii_Syrc`	Syrc	57	udhr
`ajg_Latn`	Latn	59	udhr
`ajp_Arab`	Arab	997	flores
`aka_Latn`	Latn	997	flores
`als_Latn`	Latn	1056	flores, udhr
`alt_Cyrl`	Cyrl	51	udhr
`amc_Latn`	Latn	56	udhr
`ame_Latn`	Latn	59	udhr
`amh_Ethi`	Ethi	1048	flores, udhr
`ami_Latn`	Latn	58	udhr
`amr_Latn`	Latn	61	udhr
`apc_Arab`	Arab	997	flores
`arb_Arab`	Arab	1055	flores, udhr
`arb_Latn`	Latn	997	flores
`arl_Latn`	Latn	60	udhr
`arn_Latn`	Latn	58	udhr
`ars_Arab`	Arab	997	flores
`ary_Arab`	Arab	997	flores
`arz_Arab`	Arab	997	flores
`asm_Beng`	Beng	997	flores
`ast_Latn`	Latn	1055	flores, udhr
`auc_Latn`	Latn	154	udhr
`awa_Deva`	Deva	997	flores
`ayr_Latn`	Latn	1056	flores, udhr
`azb_Arab`	Arab	997	flores
`azj_Cyrl`	Cyrl	56	udhr
`azj_Latn`	Latn	1055	flores, udhr
`bak_Cyrl`	Cyrl	997	flores
`bam_Latn`	Latn	1054	flores, udhr
`ban_Latn`	Latn	1057	flores, udhr
`bax_Latn`	Latn	58	udhr
`bba_Latn`	Latn	58	udhr
`bci_Latn`	Latn	58	udhr
`bcl_Latn`	Latn	59	udhr
`bel_Cyrl`	Cyrl	1055	flores, udhr
`bem_Latn`	Latn	1054	flores, udhr
`ben_Beng`	Beng	1055	flores, udhr
`bfa_Latn`	Latn	42	udhr
`bho_Deva`	Deva	1053	flores, udhr
`bin_Latn`	Latn	47	udhr
`bis_Latn`	Latn	59	udhr
`bjn_Arab`	Arab	997	flores
`bjn_Latn`	Latn	997	flores
`blt_Tavt`	Tavt	55	udhr
`boa_Latn`	Latn	61	udhr
`bod_Tibt`	Tibt	1057	flores, udhr
`bos_Cyrl`	Cyrl	59	udhr
`bos_Latn`	Latn	1056	flores, udhr
`bre_Latn`	Latn	59	udhr
`buc_Latn`	Latn	57	udhr
`bug_Latn`	Latn	1057	flores, udhr
`bul_Cyrl`	Cyrl	1055	flores, udhr
`bum_Latn`	Latn	58	udhr
`cab_Latn`	Latn	60	udhr
`cak_Latn`	Latn	57	udhr
`cat_Latn`	Latn	1055	flores, udhr
`cbi_Latn`	Latn	62	udhr
`cbr_Latn`	Latn	60	udhr
`cbs_Latn`	Latn	43	udhr
`cbt_Latn`	Latn	57	udhr
`cbu_Latn`	Latn	57	udhr
`ccp_Cakm`	Cakm	62	udhr
`ceb_Latn`	Latn	1056	flores, udhr
`ces_Latn`	Latn	1055	flores, udhr
`cfm_Latn`	Latn	57	udhr
`cha_Latn`	Latn	57	udhr
`chj_Latn`	Latn	59	udhr
`chk_Latn`	Latn	58	udhr
`chr_Cher`	Cher	115	udhr
`chv_Cyrl`	Cyrl	25	udhr
`cic_Latn`	Latn	60	udhr
`cjk_Latn`	Latn	1111	flores, udhr
`cjs_Cyrl`	Cyrl	59	udhr
`cjy_Hani`	Hani	58	udhr
`ckb_Arab`	Arab	997	flores
`cmn_Hani`	Hani	363	udhr
`cnh_Latn`	Latn	58	udhr
`cni_Latn`	Latn	52	udhr
`cnr_Latn`	Latn	59	udhr
`cof_Latn`	Latn	59	udhr
`cos_Latn`	Latn	59	udhr
`cot_Latn`	Latn	60	udhr
`cpu_Latn`	Latn	60	udhr
`crh_Latn`	Latn	1056	flores, udhr
`cri_Latn`	Latn	58	udhr
`crs_Latn`	Latn	58	udhr
`csa_Latn`	Latn	57	udhr
`csw_Cans`	Cans	43	udhr
`ctd_Latn`	Latn	60	udhr
`cym_Latn`	Latn	1057	flores, udhr
`dag_Latn`	Latn	57	udhr
`dan_Latn`	Latn	1058	flores, udhr
`ddn_Latn`	Latn	58	udhr
`deu_Latn`	Latn	1064	flores, udhr
`dga_Latn`	Latn	59	udhr
`dik_Latn`	Latn	997	flores
`dip_Latn`	Latn	57	udhr
`div_Thaa`	Thaa	57	udhr
`dje_Latn`	Latn	54	udhr
`duu_Latn`	Latn	57	udhr
`dyo_Latn`	Latn	55	udhr
`dyu_Latn`	Latn	1055	flores, udhr
`dzo_Tibt`	Tibt	1056	flores, udhr
`ekk_Latn`	Latn	60	udhr
`ell_Grek`	Grek	1113	flores, udhr
`emk_Latn`	Latn	58	udhr
`eng_Latn`	Latn	1055	flores, udhr
`epo_Latn`	Latn	1055	flores, udhr
`ese_Latn`	Latn	35	udhr
`est_Latn`	Latn	997	flores
`eus_Latn`	Latn	1056	flores, udhr
`eve_Cyrl`	Cyrl	49	udhr
`evn_Cyrl`	Cyrl	57	udhr
`ewe_Latn`	Latn	1056	flores, udhr
`fao_Latn`	Latn	1054	flores, udhr
`fat_Latn`	Latn	57	udhr
`fij_Latn`	Latn	1056	flores, udhr
`fin_Latn`	Latn	1058	flores, udhr
`fkv_Latn`	Latn	58	udhr
`fon_Latn`	Latn	1055	flores, udhr
`fra_Latn`	Latn	1055	flores, udhr
`fry_Latn`	Latn	58	udhr
`fuf_Adlm`	Adlm	58	udhr
`fuf_Latn`	Latn	58	udhr
`fur_Latn`	Latn	1056	flores, udhr
`fuv_Latn`	Latn	1105	flores, udhr
`fvr_Latn`	Latn	57	udhr
`gaa_Latn`	Latn	59	udhr
`gag_Latn`	Latn	59	udhr
`gan_Hani`	Hani	58	udhr
`gaz_Latn`	Latn	1057	flores, udhr
`gej_Latn`	Latn	58	udhr
`gjn_Latn`	Latn	60	udhr
`gkp_Latn`	Latn	56	udhr
`gla_Latn`	Latn	1059	flores, udhr
`gld_Cyrl`	Cyrl	59	udhr
`gle_Latn`	Latn	1053	flores, udhr
`glg_Latn`	Latn	1055	flores, udhr
`glv_Latn`	Latn	58	udhr
`grn_Latn`	Latn	997	flores
`gsw_Latn`	Latn	59	udhr
`guc_Latn`	Latn	58	udhr
`gug_Latn`	Latn	49	udhr
`guj_Gujr`	Gujr	1055	flores, udhr
`guk_Latn`	Latn	55	udhr
`guu_Latn`	Latn	58	udhr
`gyr_Latn`	Latn	49	udhr
`hak_Hani`	Hani	57	udhr
`hat_Latn`	Latn	1114	flores, udhr
`hau_Latn`	Latn	1152	flores, udhr
`haw_Latn`	Latn	58	udhr
`hea_Latn`	Latn	58	udhr
`heb_Hebr`	Hebr	1055	flores, udhr
`hil_Latn`	Latn	58	udhr
`hin_Deva`	Deva	1057	flores, udhr
`hlt_Latn`	Latn	55	udhr
`hms_Latn`	Latn	59	udhr
`hne_Deva`	Deva	997	flores
`hni_Latn`	Latn	57	udhr
`hnj_Latn`	Latn	58	udhr
`hns_Latn`	Latn	59	udhr
`hrv_Latn`	Latn	1055	flores, udhr
`hsb_Latn`	Latn	59	udhr
`hsn_Hani`	Hani	58	udhr
`hun_Latn`	Latn	1056	flores, udhr
`hus_Latn`	Latn	175	udhr
`huu_Latn`	Latn	58	udhr
`hye_Armn`	Armn	1068	flores, udhr
`ibb_Latn`	Latn	67	udhr
`ibo_Latn`	Latn	1055	flores, udhr
`ido_Latn`	Latn	55	udhr
`idu_Latn`	Latn	51	udhr
`iii_Yiii`	Yiii	57	udhr
`ijs_Latn`	Latn	50	udhr
`ike_Cans`	Cans	44	udhr
`ilo_Latn`	Latn	1055	flores, udhr
`ina_Latn`	Latn	58	udhr
`ind_Latn`	Latn	1057	flores, udhr
`isl_Latn`	Latn	1055	flores, udhr
`ita_Latn`	Latn	1055	flores, udhr
`jav_Java`	Java	62	udhr
`jav_Latn`	Latn	1057	flores, udhr
`jiv_Latn`	Latn	60	udhr
`jpn_Jpan`	Jpan	1140	flores, udhr
`kaa_Cyrl`	Cyrl	58	udhr
`kab_Latn`	Latn	1055	flores, udhr
`kac_Latn`	Latn	997	flores
`kal_Latn`	Latn	58	udhr
`kam_Latn`	Latn	997	flores
`kan_Knda`	Knda	1055	flores, udhr
`kas_Arab`	Arab	997	flores
`kas_Deva`	Deva	997	flores
`kat_Geor`	Geor	1055	flores, udhr
`kaz_Cyrl`	Cyrl	1055	flores, udhr
`kbd_Cyrl`	Cyrl	55	udhr
`kbp_Latn`	Latn	1056	flores, udhr
`kbr_Latn`	Latn	59	udhr
`kde_Latn`	Latn	51	udhr
`kdh_Latn`	Latn	59	udhr
`kea_Latn`	Latn	1055	flores, udhr
`kek_Latn`	Latn	57	udhr
`kha_Latn`	Latn	58	udhr
`khk_Cyrl`	Cyrl	1054	flores, udhr
`khm_Khmr`	Khmr	1056	flores, udhr
`kik_Latn`	Latn	997	flores
`kin_Latn`	Latn	1056	flores, udhr
`kir_Cyrl`	Cyrl	1055	flores, udhr
`kjh_Cyrl`	Cyrl	57	udhr
`kkh_Lana`	Lana	51	udhr
`kmb_Latn`	Latn	1113	flores, udhr
`kmr_Latn`	Latn	1055	flores, udhr
`knc_Arab`	Arab	997	flores
`knc_Latn`	Latn	1061	flores, udhr
`kng_Latn`	Latn	115	udhr
`koi_Cyrl`	Cyrl	57	udhr
`kon_Latn`	Latn	997	flores
`koo_Latn`	Latn	54	udhr
`kor_Hang`	Hang	1055	flores, udhr
`kqn_Latn`	Latn	59	udhr
`kqs_Latn`	Latn	58	udhr
`kri_Latn`	Latn	58	udhr
`krl_Latn`	Latn	183	udhr
`ktu_Latn`	Latn	118	udhr
`kwi_Latn`	Latn	145	udhr
`lad_Latn`	Latn	58	udhr
`lao_Laoo`	Laoo	1054	flores, udhr
`lat_Latn`	Latn	118	udhr
`lia_Latn`	Latn	45	udhr
`lij_Latn`	Latn	1055	flores, udhr
`lim_Latn`	Latn	997	flores
`lin_Latn`	Latn	1107	flores, udhr
`lit_Latn`	Latn	1055	flores, udhr
`lld_Latn`	Latn	59	udhr
`lmo_Latn`	Latn	997	flores
`lns_Latn`	Latn	57	udhr
`lob_Latn`	Latn	58	udhr
`lot_Latn`	Latn	140	udhr
`loz_Latn`	Latn	58	udhr
`ltg_Latn`	Latn	997	flores
`ltz_Latn`	Latn	1056	flores, udhr
`lua_Latn`	Latn	1055	flores, udhr
`lue_Latn`	Latn	58	udhr
`lug_Latn`	Latn	1046	flores, udhr
`lun_Latn`	Latn	56	udhr
`luo_Latn`	Latn	997	flores
`lus_Latn`	Latn	1055	flores, udhr
`lvs_Latn`	Latn	1112	flores, udhr
`mad_Latn`	Latn	60	udhr
`mag_Deva`	Deva	1050	flores, udhr
`mah_Latn`	Latn	58	udhr
`mai_Deva`	Deva	1055	flores, udhr
`mal_Mlym`	Mlym	1097	flores, udhr
`mam_Latn`	Latn	58	udhr
`mar_Deva`	Deva	1056	flores, udhr
`maz_Latn`	Latn	59	udhr
`mcd_Latn`	Latn	60	udhr
`mcf_Latn`	Latn	62	udhr
`men_Latn`	Latn	59	udhr
`mfq_Latn`	Latn	58	udhr
`mic_Latn`	Latn	59	udhr
`min_Arab`	Arab	997	flores
`min_Latn`	Latn	1056	flores, udhr
`miq_Latn`	Latn	51	udhr
`mkd_Cyrl`	Cyrl	1055	flores, udhr
`mlt_Latn`	Latn	1056	flores, udhr
`mni_Beng`	Beng	997	flores
`mnw_Mymr`	Mymr	56	udhr
`mor_Latn`	Latn	59	udhr
`mos_Latn`	Latn	1053	flores, udhr
`mri_Latn`	Latn	1054	flores, udhr
`mto_Latn`	Latn	51	udhr
`mxi_Latn`	Latn	58	udhr
`mxv_Latn`	Latn	58	udhr
`mya_Mymr`	Mymr	1053	flores, udhr
`mzi_Latn`	Latn	58	udhr
`nan_Hani`	Hani	58	udhr
`nav_Latn`	Latn	59	udhr
`nba_Latn`	Latn	59	udhr
`nbl_Latn`	Latn	58	udhr
`ndo_Latn`	Latn	58	udhr
`nds_Latn`	Latn	58	udhr
`nhn_Latn`	Latn	58	udhr
`nio_Cyrl`	Cyrl	51	udhr
`niu_Latn`	Latn	58	udhr
`niv_Cyrl`	Cyrl	56	udhr
`njo_Latn`	Latn	64	udhr
`nku_Latn`	Latn	52	udhr
`nld_Latn`	Latn	1055	flores, udhr
`nno_Latn`	Latn	1055	flores, udhr
`nob_Latn`	Latn	1059	flores, udhr
`not_Latn`	Latn	59	udhr
`npi_Deva`	Deva	1052	flores, udhr
`nso_Latn`	Latn	1055	flores, udhr
`nus_Latn`	Latn	997	flores
`nya_Latn`	Latn	1113	flores, udhr
`nym_Latn`	Latn	49	udhr
`nyn_Latn`	Latn	50	udhr
`nzi_Latn`	Latn	59	udhr
`oaa_Cyrl`	Cyrl	58	udhr
`oci_Latn`	Latn	1395	flores, udhr
`ojb_Cans`	Cans	44	udhr
`oki_Latn`	Latn	59	udhr
`orh_Latn`	Latn	59	udhr
`ory_Orya`	Orya	997	flores
`oss_Cyrl`	Cyrl	44	udhr
`ote_Latn`	Latn	59	udhr
`pag_Latn`	Latn	997	flores
`pam_Latn`	Latn	66	udhr
`pan_Guru`	Guru	1055	flores, udhr
`pap_Latn`	Latn	1055	flores, udhr
`pau_Latn`	Latn	57	udhr
`pbb_Latn`	Latn	56	udhr
`pbt_Arab`	Arab	997	flores
`pbu_Arab`	Arab	59	udhr
`pcd_Latn`	Latn	58	udhr
`pcm_Latn`	Latn	56	udhr
`pes_Arab`	Arab	1055	flores, udhr
`pis_Latn`	Latn	57	udhr
`piu_Latn`	Latn	57	udhr
`plt_Latn`	Latn	1050	flores, udhr
`pnb_Arab`	Arab	61	udhr
`pol_Latn`	Latn	1056	flores, udhr
`pon_Latn`	Latn	57	udhr
`por_Latn`	Latn	1112	flores, udhr
`pov_Latn`	Latn	115	udhr
`ppl_Latn`	Latn	59	udhr
`prs_Arab`	Arab	1053	flores, udhr
`quc_Latn`	Latn	58	udhr
`que_Latn`	Latn	58	udhr
`qug_Latn`	Latn	182	udhr
`quh_Latn`	Latn	57	udhr
`quy_Latn`	Latn	1057	flores, udhr
`quz_Latn`	Latn	57	udhr
`qva_Latn`	Latn	59	udhr
`qvc_Latn`	Latn	58	udhr
`qvh_Latn`	Latn	58	udhr
`qvm_Latn`	Latn	60	udhr
`qvn_Latn`	Latn	57	udhr
`qwh_Latn`	Latn	59	udhr
`qxn_Latn`	Latn	57	udhr
`qxu_Latn`	Latn	54	udhr
`rar_Latn`	Latn	59	udhr
`rgn_Latn`	Latn	58	udhr
`rmn_Latn`	Latn	116	udhr
`roh_Latn`	Latn	399	udhr
`ron_Latn`	Latn	1094	flores, udhr
`run_Latn`	Latn	1055	flores, udhr
`rup_Latn`	Latn	57	udhr
`rus_Cyrl`	Cyrl	1055	flores, udhr
`sag_Latn`	Latn	1056	flores, udhr
`sah_Cyrl`	Cyrl	57	udhr
`san_Deva`	Deva	1052	flores, udhr
`san_Gran`	Gran	51	udhr
`sat_Olck`	Olck	997	flores
`scn_Latn`	Latn	997	flores
`sco_Latn`	Latn	58	udhr
`sey_Latn`	Latn	61	udhr
`shk_Latn`	Latn	55	udhr
`shn_Mymr`	Mymr	1053	flores, udhr
`shp_Latn`	Latn	62	udhr
`sid_Latn`	Latn	57	udhr
`sin_Sinh`	Sinh	1056	flores, udhr
`skr_Arab`	Arab	63	udhr
`slk_Latn`	Latn	1053	flores, udhr
`slr_Latn`	Latn	58	udhr
`slv_Latn`	Latn	1056	flores, udhr
`sme_Latn`	Latn	58	udhr
`smo_Latn`	Latn	1060	flores, udhr
`sna_Latn`	Latn	1057	flores, udhr
`snd_Arab`	Arab	997	flores
`snk_Latn`	Latn	58	udhr
`snn_Latn`	Latn	136	udhr
`som_Latn`	Latn	1054	flores, udhr
`sot_Latn`	Latn	1054	flores, udhr
`spa_Latn`	Latn	1075	flores, udhr
`src_Latn`	Latn	59	udhr
`srd_Latn`	Latn	997	flores
`srp_Cyrl`	Cyrl	1056	flores, udhr
`srp_Latn`	Latn	59	udhr
`srr_Latn`	Latn	58	udhr
`ssw_Latn`	Latn	1057	flores, udhr
`suk_Latn`	Latn	58	udhr
`sun_Latn`	Latn	1063	flores, udhr
`sus_Latn`	Latn	58	udhr
`swb_Latn`	Latn	58	udhr
`swe_Latn`	Latn	1058	flores, udhr
`swh_Latn`	Latn	1055	flores, udhr
`szl_Latn`	Latn	997	flores
`tah_Latn`	Latn	57	udhr
`tam_Taml`	Taml	1066	flores, udhr
`taq_Latn`	Latn	997	flores
`taq_Tfng`	Tfng	997	flores
`tat_Cyrl`	Cyrl	1054	flores, udhr
`tbz_Latn`	Latn	60	udhr
`tca_Latn`	Latn	60	udhr
`tdt_Latn`	Latn	57	udhr
`tel_Telu`	Telu	1055	flores, udhr
`tem_Latn`	Latn	55	udhr
`tet_Latn`	Latn	58	udhr
`tgk_Cyrl`	Cyrl	1054	flores, udhr
`tgl_Latn`	Latn	1057	flores, udhr
`tha_Thai`	Thai	1111	flores, udhr
`tir_Ethi`	Ethi	1055	flores, udhr
`tiv_Latn`	Latn	67	udhr
`tly_Latn`	Latn	59	udhr
`tob_Latn`	Latn	59	udhr
`toi_Latn`	Latn	57	udhr
`toj_Latn`	Latn	57	udhr
`ton_Latn`	Latn	60	udhr
`top_Latn`	Latn	59	udhr
`tpi_Latn`	Latn	1055	flores, udhr
`tsn_Latn`	Latn	1055	flores, udhr
`tso_Latn`	Latn	1111	flores, udhr
`tsz_Latn`	Latn	59	udhr
`tuk_Cyrl`	Cyrl	57	udhr
`tuk_Latn`	Latn	1056	flores, udhr
`tum_Latn`	Latn	997	flores
`tur_Latn`	Latn	1056	flores, udhr
`twi_Latn`	Latn	1113	flores, udhr
`tyv_Cyrl`	Cyrl	58	udhr
`tzh_Latn`	Latn	25	udhr
`tzm_Tfng`	Tfng	998	flores, udhr
`tzo_Latn`	Latn	58	udhr
`udu_Latn`	Latn	60	udhr
`uig_Arab`	Arab	1055	flores, udhr
`uig_Latn`	Latn	59	udhr
`ukr_Cyrl`	Cyrl	1055	flores, udhr
`umb_Latn`	Latn	1104	flores, udhr
`ura_Latn`	Latn	56	udhr
`urd_Arab`	Arab	1105	flores, udhr
`uzn_Cyrl`	Cyrl	58	udhr
`uzn_Latn`	Latn	1055	flores, udhr
`vai_Vaii`	Vaii	61	udhr
`vec_Latn`	Latn	1055	flores, udhr
`ven_Latn`	Latn	89	udhr
`vep_Latn`	Latn	58	udhr
`vie_Latn`	Latn	1057	flores, udhr
`vmw_Latn`	Latn	58	udhr
`war_Latn`	Latn	1057	flores, udhr
`wln_Latn`	Latn	59	udhr
`wol_Latn`	Latn	1057	flores, udhr
`wuu_Hani`	Hani	58	udhr
`wwa_Latn`	Latn	39	udhr
`xho_Latn`	Latn	1055	flores, udhr
`xsm_Latn`	Latn	59	udhr
`yad_Latn`	Latn	55	udhr
`yao_Latn`	Latn	52	udhr
`yap_Latn`	Latn	58	udhr
`ydd_Hebr`	Hebr	1056	flores, udhr
`ykg_Cyrl`	Cyrl	51	udhr
`yor_Latn`	Latn	1054	flores, udhr
`yrk_Cyrl`	Cyrl	55	udhr
`yua_Latn`	Latn	59	udhr
`yue_Hani`	Hani	59	udhr
`yue_Hant`	Hant	997	flores
`zam_Latn`	Latn	60	udhr
`zdj_Latn`	Latn	50	udhr
`zgh_Tfng`	Tfng	61	udhr
`zho_Hans`	Hans	997	flores
`zho_Hant`	Hant	997	flores
`zlm_Arab`	Arab	24	udhr
`zlm_Latn`	Latn	58	udhr
`zro_Latn`	Latn	56	udhr
`zsm_Latn`	Latn	997	flores
`ztu_Latn`	Latn	58	udhr
`zul_Latn`	Latn	1055	flores, udhr
`zyb_Latn`	Latn	57	udhr

Citation

Built with Tessera -- span-level multi-label language identification. Repository: NagaYu/tessera-mixed-pages.