allenai/c4

1---2pretty_name: C43annotations_creators:4- no-annotation5language_creators:6- found7language:8- af9- am10- ar11- az12- be13- bg14- bn15- ca16- ceb17- co18- cs19- cy20- da21- de22- el23- en24- eo25- es26- et27- eu28- fa29- fi30- fil31- fr32- fy33- ga34- gd35- gl36- gu37- ha38- haw39- he40- hi41- hmn42- ht43- hu44- hy45- id46- ig47- is48- it49- iw50- ja51- jv52- ka53- kk54- km55- kn56- ko57- ku58- ky59- la60- lb61- lo62- lt63- lv64- mg65- mi66- mk67- ml68- mn69- mr70- ms71- mt72- my73- ne74- nl75- 'no'76- ny77- pa78- pl79- ps80- pt81- ro82- ru83- sd84- si85- sk86- sl87- sm88- sn89- so90- sq91- sr92- st93- su94- sv95- sw96- ta97- te98- tg99- th100- tr101- uk102- und103- ur104- uz105- vi106- xh107- yi108- yo109- zh110- zu111language_bcp47:112- bg-Latn113- el-Latn114- hi-Latn115- ja-Latn116- ru-Latn117- zh-Latn118license:119- odc-by120multilinguality:121- multilingual122size_categories:123- n<1K124- 1K<n<10K125- 10K<n<100K126- 100K<n<1M127- 1M<n<10M128- 10M<n<100M129- 100M<n<1B130- 1B<n<10B131source_datasets:132- original133task_categories:134- text-generation135- fill-mask136task_ids:137- language-modeling138- masked-language-modeling139paperswithcode_id: c4140dataset_info:141- config_name: en142  features:143  - name: text144    dtype: string145  - name: timestamp146    dtype: string147  - name: url148    dtype: string149  splits:150  - name: train151    num_bytes: 828589180707152    num_examples: 364868892153  - name: validation154    num_bytes: 825767266155    num_examples: 364608156  download_size: 326778635540157  dataset_size: 1657178361414158- config_name: en.noblocklist159  features:160  - name: text161    dtype: string162  - name: timestamp163    dtype: string164  - name: url165    dtype: string166  splits:167  - name: train168    num_bytes: 1029628201361169    num_examples: 393391519170  - name: validation171    num_bytes: 1025606012172    num_examples: 393226173  download_size: 406611392434174  dataset_size: 2059256402722175- config_name: realnewslike176  features:177  - name: text178    dtype: string179  - name: timestamp180    dtype: string181  - name: url182    dtype: string183  splits:184  - name: train185    num_bytes: 38165657946186    num_examples: 13799838187  - name: validation188    num_bytes: 37875873189    num_examples: 13863190  download_size: 15419740744191  dataset_size: 76331315892192- config_name: en.noclean193  features:194  - name: text195    dtype: string196  - name: timestamp197    dtype: string198  - name: url199    dtype: string200  splits:201  - name: train202    num_bytes: 6715509699938203    num_examples: 1063805381204  - name: validation205    num_bytes: 6706356913206    num_examples: 1065029207  download_size: 2430376268625208  dataset_size: 6722216056851209configs:210- config_name: en211  data_files:212  - split: train213    path: en/c4-train.*.json.gz214  - split: validation215    path: en/c4-validation.*.json.gz216- config_name: en.noblocklist217  data_files:218  - split: train219    path: en.noblocklist/c4-train.*.json.gz220  - split: validation221    path: en.noblocklist/c4-validation.*.json.gz222- config_name: en.noclean223  data_files:224  - split: train225    path: en.noclean/c4-train.*.json.gz226  - split: validation227    path: en.noclean/c4-validation.*.json.gz228- config_name: realnewslike229  data_files:230  - split: train231    path: realnewslike/c4-train.*.json.gz232  - split: validation233    path: realnewslike/c4-validation.*.json.gz234- config_name: multilingual235  data_files:236  - split: train237    path:238    - multilingual/c4-af.*.json.gz239    - multilingual/c4-am.*.json.gz240    - multilingual/c4-ar.*.json.gz241    - multilingual/c4-az.*.json.gz242    - multilingual/c4-be.*.json.gz243    - multilingual/c4-bg.*.json.gz244    - multilingual/c4-bg-Latn.*.json.gz245    - multilingual/c4-bn.*.json.gz246    - multilingual/c4-ca.*.json.gz247    - multilingual/c4-ceb.*.json.gz248    - multilingual/c4-co.*.json.gz249    - multilingual/c4-cs.*.json.gz250    - multilingual/c4-cy.*.json.gz251    - multilingual/c4-da.*.json.gz252    - multilingual/c4-de.*.json.gz253    - multilingual/c4-el.*.json.gz254    - multilingual/c4-el-Latn.*.json.gz255    - multilingual/c4-en.*.json.gz256    - multilingual/c4-eo.*.json.gz257    - multilingual/c4-es.*.json.gz258    - multilingual/c4-et.*.json.gz259    - multilingual/c4-eu.*.json.gz260    - multilingual/c4-fa.*.json.gz261    - multilingual/c4-fi.*.json.gz262    - multilingual/c4-fil.*.json.gz263    - multilingual/c4-fr.*.json.gz264    - multilingual/c4-fy.*.json.gz265    - multilingual/c4-ga.*.json.gz266    - multilingual/c4-gd.*.json.gz267    - multilingual/c4-gl.*.json.gz268    - multilingual/c4-gu.*.json.gz269    - multilingual/c4-ha.*.json.gz270    - multilingual/c4-haw.*.json.gz271    - multilingual/c4-hi.*.json.gz272    - multilingual/c4-hi-Latn.*.json.gz273    - multilingual/c4-hmn.*.json.gz274    - multilingual/c4-ht.*.json.gz275    - multilingual/c4-hu.*.json.gz276    - multilingual/c4-hy.*.json.gz277    - multilingual/c4-id.*.json.gz278    - multilingual/c4-ig.*.json.gz279    - multilingual/c4-is.*.json.gz280    - multilingual/c4-it.*.json.gz281    - multilingual/c4-iw.*.json.gz282    - multilingual/c4-ja.*.json.gz283    - multilingual/c4-ja-Latn.*.json.gz284    - multilingual/c4-jv.*.json.gz285    - multilingual/c4-ka.*.json.gz286    - multilingual/c4-kk.*.json.gz287    - multilingual/c4-km.*.json.gz288    - multilingual/c4-kn.*.json.gz289    - multilingual/c4-ko.*.json.gz290    - multilingual/c4-ku.*.json.gz291    - multilingual/c4-ky.*.json.gz292    - multilingual/c4-la.*.json.gz293    - multilingual/c4-lb.*.json.gz294    - multilingual/c4-lo.*.json.gz295    - multilingual/c4-lt.*.json.gz296    - multilingual/c4-lv.*.json.gz297    - multilingual/c4-mg.*.json.gz298    - multilingual/c4-mi.*.json.gz299    - multilingual/c4-mk.*.json.gz300    - multilingual/c4-ml.*.json.gz301    - multilingual/c4-mn.*.json.gz302    - multilingual/c4-mr.*.json.gz303    - multilingual/c4-ms.*.json.gz304    - multilingual/c4-mt.*.json.gz305    - multilingual/c4-my.*.json.gz306    - multilingual/c4-ne.*.json.gz307    - multilingual/c4-nl.*.json.gz308    - multilingual/c4-no.*.json.gz309    - multilingual/c4-ny.*.json.gz310    - multilingual/c4-pa.*.json.gz311    - multilingual/c4-pl.*.json.gz312    - multilingual/c4-ps.*.json.gz313    - multilingual/c4-pt.*.json.gz314    - multilingual/c4-ro.*.json.gz315    - multilingual/c4-ru.*.json.gz316    - multilingual/c4-ru-Latn.*.json.gz317    - multilingual/c4-sd.*.json.gz318    - multilingual/c4-si.*.json.gz319    - multilingual/c4-sk.*.json.gz320    - multilingual/c4-sl.*.json.gz321    - multilingual/c4-sm.*.json.gz322    - multilingual/c4-sn.*.json.gz323    - multilingual/c4-so.*.json.gz324    - multilingual/c4-sq.*.json.gz325    - multilingual/c4-sr.*.json.gz326    - multilingual/c4-st.*.json.gz327    - multilingual/c4-su.*.json.gz328    - multilingual/c4-sv.*.json.gz329    - multilingual/c4-sw.*.json.gz330    - multilingual/c4-ta.*.json.gz331    - multilingual/c4-te.*.json.gz332    - multilingual/c4-tg.*.json.gz333    - multilingual/c4-th.*.json.gz334    - multilingual/c4-tr.*.json.gz335    - multilingual/c4-uk.*.json.gz336    - multilingual/c4-und.*.json.gz337    - multilingual/c4-ur.*.json.gz338    - multilingual/c4-uz.*.json.gz339    - multilingual/c4-vi.*.json.gz340    - multilingual/c4-xh.*.json.gz341    - multilingual/c4-yi.*.json.gz342    - multilingual/c4-yo.*.json.gz343    - multilingual/c4-zh.*.json.gz344    - multilingual/c4-zh-Latn.*.json.gz345    - multilingual/c4-zu.*.json.gz346  - split: validation347    path:348    - multilingual/c4-af-validation.*.json.gz349    - multilingual/c4-am-validation.*.json.gz350    - multilingual/c4-ar-validation.*.json.gz351    - multilingual/c4-az-validation.*.json.gz352    - multilingual/c4-be-validation.*.json.gz353    - multilingual/c4-bg-validation.*.json.gz354    - multilingual/c4-bg-Latn-validation.*.json.gz355    - multilingual/c4-bn-validation.*.json.gz356    - multilingual/c4-ca-validation.*.json.gz357    - multilingual/c4-ceb-validation.*.json.gz358    - multilingual/c4-co-validation.*.json.gz359    - multilingual/c4-cs-validation.*.json.gz360    - multilingual/c4-cy-validation.*.json.gz361    - multilingual/c4-da-validation.*.json.gz362    - multilingual/c4-de-validation.*.json.gz363    - multilingual/c4-el-validation.*.json.gz364    - multilingual/c4-el-Latn-validation.*.json.gz365    - multilingual/c4-en-validation.*.json.gz366    - multilingual/c4-eo-validation.*.json.gz367    - multilingual/c4-es-validation.*.json.gz368    - multilingual/c4-et-validation.*.json.gz369    - multilingual/c4-eu-validation.*.json.gz370    - multilingual/c4-fa-validation.*.json.gz371    - multilingual/c4-fi-validation.*.json.gz372    - multilingual/c4-fil-validation.*.json.gz373    - multilingual/c4-fr-validation.*.json.gz374    - multilingual/c4-fy-validation.*.json.gz375    - multilingual/c4-ga-validation.*.json.gz376    - multilingual/c4-gd-validation.*.json.gz377    - multilingual/c4-gl-validation.*.json.gz378    - multilingual/c4-gu-validation.*.json.gz379    - multilingual/c4-ha-validation.*.json.gz380    - multilingual/c4-haw-validation.*.json.gz381    - multilingual/c4-hi-validation.*.json.gz382    - multilingual/c4-hi-Latn-validation.*.json.gz383    - multilingual/c4-hmn-validation.*.json.gz384    - multilingual/c4-ht-validation.*.json.gz385    - multilingual/c4-hu-validation.*.json.gz386    - multilingual/c4-hy-validation.*.json.gz387    - multilingual/c4-id-validation.*.json.gz388    - multilingual/c4-ig-validation.*.json.gz389    - multilingual/c4-is-validation.*.json.gz390    - multilingual/c4-it-validation.*.json.gz391    - multilingual/c4-iw-validation.*.json.gz392    - multilingual/c4-ja-validation.*.json.gz393    - multilingual/c4-ja-Latn-validation.*.json.gz394    - multilingual/c4-jv-validation.*.json.gz395    - multilingual/c4-ka-validation.*.json.gz396    - multilingual/c4-kk-validation.*.json.gz397    - multilingual/c4-km-validation.*.json.gz398    - multilingual/c4-kn-validation.*.json.gz399    - multilingual/c4-ko-validation.*.json.gz400    - multilingual/c4-ku-validation.*.json.gz401    - multilingual/c4-ky-validation.*.json.gz402    - multilingual/c4-la-validation.*.json.gz403    - multilingual/c4-lb-validation.*.json.gz404    - multilingual/c4-lo-validation.*.json.gz405    - multilingual/c4-lt-validation.*.json.gz406    - multilingual/c4-lv-validation.*.json.gz407    - multilingual/c4-mg-validation.*.json.gz408    - multilingual/c4-mi-validation.*.json.gz409    - multilingual/c4-mk-validation.*.json.gz410    - multilingual/c4-ml-validation.*.json.gz411    - multilingual/c4-mn-validation.*.json.gz412    - multilingual/c4-mr-validation.*.json.gz413    - multilingual/c4-ms-validation.*.json.gz414    - multilingual/c4-mt-validation.*.json.gz415    - multilingual/c4-my-validation.*.json.gz416    - multilingual/c4-ne-validation.*.json.gz417    - multilingual/c4-nl-validation.*.json.gz418    - multilingual/c4-no-validation.*.json.gz419    - multilingual/c4-ny-validation.*.json.gz420    - multilingual/c4-pa-validation.*.json.gz421    - multilingual/c4-pl-validation.*.json.gz422    - multilingual/c4-ps-validation.*.json.gz423    - multilingual/c4-pt-validation.*.json.gz424    - multilingual/c4-ro-validation.*.json.gz425    - multilingual/c4-ru-validation.*.json.gz426    - multilingual/c4-ru-Latn-validation.*.json.gz427    - multilingual/c4-sd-validation.*.json.gz428    - multilingual/c4-si-validation.*.json.gz429    - multilingual/c4-sk-validation.*.json.gz430    - multilingual/c4-sl-validation.*.json.gz431    - multilingual/c4-sm-validation.*.json.gz432    - multilingual/c4-sn-validation.*.json.gz433    - multilingual/c4-so-validation.*.json.gz434    - multilingual/c4-sq-validation.*.json.gz435    - multilingual/c4-sr-validation.*.json.gz436    - multilingual/c4-st-validation.*.json.gz437    - multilingual/c4-su-validation.*.json.gz438    - multilingual/c4-sv-validation.*.json.gz439    - multilingual/c4-sw-validation.*.json.gz440    - multilingual/c4-ta-validation.*.json.gz441    - multilingual/c4-te-validation.*.json.gz442    - multilingual/c4-tg-validation.*.json.gz443    - multilingual/c4-th-validation.*.json.gz444    - multilingual/c4-tr-validation.*.json.gz445    - multilingual/c4-uk-validation.*.json.gz446    - multilingual/c4-und-validation.*.json.gz447    - multilingual/c4-ur-validation.*.json.gz448    - multilingual/c4-uz-validation.*.json.gz449    - multilingual/c4-vi-validation.*.json.gz450    - multilingual/c4-xh-validation.*.json.gz451    - multilingual/c4-yi-validation.*.json.gz452    - multilingual/c4-yo-validation.*.json.gz453    - multilingual/c4-zh-validation.*.json.gz454    - multilingual/c4-zh-Latn-validation.*.json.gz455    - multilingual/c4-zu-validation.*.json.gz456- config_name: af457  data_files:458  - split: train459    path: multilingual/c4-af.*.json.gz460  - split: validation461    path: multilingual/c4-af-validation.*.json.gz462- config_name: am463  data_files:464  - split: train465    path: multilingual/c4-am.*.json.gz466  - split: validation467    path: multilingual/c4-am-validation.*.json.gz468- config_name: ar469  data_files:470  - split: train471    path: multilingual/c4-ar.*.json.gz472  - split: validation473    path: multilingual/c4-ar-validation.*.json.gz474- config_name: az475  data_files:476  - split: train477    path: multilingual/c4-az.*.json.gz478  - split: validation479    path: multilingual/c4-az-validation.*.json.gz480- config_name: be481  data_files:482  - split: train483    path: multilingual/c4-be.*.json.gz484  - split: validation485    path: multilingual/c4-be-validation.*.json.gz486- config_name: bg487  data_files:488  - split: train489    path: multilingual/c4-bg.*.json.gz490  - split: validation491    path: multilingual/c4-bg-validation.*.json.gz492- config_name: bg-Latn493  data_files:494  - split: train495    path: multilingual/c4-bg-Latn.*.json.gz496  - split: validation497    path: multilingual/c4-bg-Latn-validation.*.json.gz498- config_name: bn499  data_files:500  - split: train501    path: multilingual/c4-bn.*.json.gz502  - split: validation503    path: multilingual/c4-bn-validation.*.json.gz504- config_name: ca505  data_files:506  - split: train507    path: multilingual/c4-ca.*.json.gz508  - split: validation509    path: multilingual/c4-ca-validation.*.json.gz510- config_name: ceb511  data_files:512  - split: train513    path: multilingual/c4-ceb.*.json.gz514  - split: validation515    path: multilingual/c4-ceb-validation.*.json.gz516- config_name: co517  data_files:518  - split: train519    path: multilingual/c4-co.*.json.gz520  - split: validation521    path: multilingual/c4-co-validation.*.json.gz522- config_name: cs523  data_files:524  - split: train525    path: multilingual/c4-cs.*.json.gz526  - split: validation527    path: multilingual/c4-cs-validation.*.json.gz528- config_name: cy529  data_files:530  - split: train531    path: multilingual/c4-cy.*.json.gz532  - split: validation533    path: multilingual/c4-cy-validation.*.json.gz534- config_name: da535  data_files:536  - split: train537    path: multilingual/c4-da.*.json.gz538  - split: validation539    path: multilingual/c4-da-validation.*.json.gz540- config_name: de541  data_files:542  - split: train543    path: multilingual/c4-de.*.json.gz544  - split: validation545    path: multilingual/c4-de-validation.*.json.gz546- config_name: el547  data_files:548  - split: train549    path: multilingual/c4-el.*.json.gz550  - split: validation551    path: multilingual/c4-el-validation.*.json.gz552- config_name: el-Latn553  data_files:554  - split: train555    path: multilingual/c4-el-Latn.*.json.gz556  - split: validation557    path: multilingual/c4-el-Latn-validation.*.json.gz558- config_name: en-multi559  data_files:560  - split: train561    path: multilingual/c4-en.*.json.gz562  - split: validation563    path: multilingual/c4-en-validation.*.json.gz564- config_name: eo565  data_files:566  - split: train567    path: multilingual/c4-eo.*.json.gz568  - split: validation569    path: multilingual/c4-eo-validation.*.json.gz570- config_name: es571  data_files:572  - split: train573    path: multilingual/c4-es.*.json.gz574  - split: validation575    path: multilingual/c4-es-validation.*.json.gz576- config_name: et577  data_files:578  - split: train579    path: multilingual/c4-et.*.json.gz580  - split: validation581    path: multilingual/c4-et-validation.*.json.gz582- config_name: eu583  data_files:584  - split: train585    path: multilingual/c4-eu.*.json.gz586  - split: validation587    path: multilingual/c4-eu-validation.*.json.gz588- config_name: fa589  data_files:590  - split: train591    path: multilingual/c4-fa.*.json.gz592  - split: validation593    path: multilingual/c4-fa-validation.*.json.gz594- config_name: fi595  data_files:596  - split: train597    path: multilingual/c4-fi.*.json.gz598  - split: validation599    path: multilingual/c4-fi-validation.*.json.gz600- config_name: fil601  data_files:602  - split: train603    path: multilingual/c4-fil.*.json.gz604  - split: validation605    path: multilingual/c4-fil-validation.*.json.gz606- config_name: fr607  data_files:608  - split: train609    path: multilingual/c4-fr.*.json.gz610  - split: validation611    path: multilingual/c4-fr-validation.*.json.gz612- config_name: fy613  data_files:614  - split: train615    path: multilingual/c4-fy.*.json.gz616  - split: validation617    path: multilingual/c4-fy-validation.*.json.gz618- config_name: ga619  data_files:620  - split: train621    path: multilingual/c4-ga.*.json.gz622  - split: validation623    path: multilingual/c4-ga-validation.*.json.gz624- config_name: gd625  data_files:626  - split: train627    path: multilingual/c4-gd.*.json.gz628  - split: validation629    path: multilingual/c4-gd-validation.*.json.gz630- config_name: gl631  data_files:632  - split: train633    path: multilingual/c4-gl.*.json.gz634  - split: validation635    path: multilingual/c4-gl-validation.*.json.gz636- config_name: gu637  data_files:638  - split: train639    path: multilingual/c4-gu.*.json.gz640  - split: validation641    path: multilingual/c4-gu-validation.*.json.gz642- config_name: ha643  data_files:644  - split: train645    path: multilingual/c4-ha.*.json.gz646  - split: validation647    path: multilingual/c4-ha-validation.*.json.gz648- config_name: haw649  data_files:650  - split: train651    path: multilingual/c4-haw.*.json.gz652  - split: validation653    path: multilingual/c4-haw-validation.*.json.gz654- config_name: hi655  data_files:656  - split: train657    path: multilingual/c4-hi.*.json.gz658  - split: validation659    path: multilingual/c4-hi-validation.*.json.gz660- config_name: hi-Latn661  data_files:662  - split: train663    path: multilingual/c4-hi-Latn.*.json.gz664  - split: validation665    path: multilingual/c4-hi-Latn-validation.*.json.gz666- config_name: hmn667  data_files:668  - split: train669    path: multilingual/c4-hmn.*.json.gz670  - split: validation671    path: multilingual/c4-hmn-validation.*.json.gz672- config_name: ht673  data_files:674  - split: train675    path: multilingual/c4-ht.*.json.gz676  - split: validation677    path: multilingual/c4-ht-validation.*.json.gz678- config_name: hu679  data_files:680  - split: train681    path: multilingual/c4-hu.*.json.gz682  - split: validation683    path: multilingual/c4-hu-validation.*.json.gz684- config_name: hy685  data_files:686  - split: train687    path: multilingual/c4-hy.*.json.gz688  - split: validation689    path: multilingual/c4-hy-validation.*.json.gz690- config_name: id691  data_files:692  - split: train693    path: multilingual/c4-id.*.json.gz694  - split: validation695    path: multilingual/c4-id-validation.*.json.gz696- config_name: ig697  data_files:698  - split: train699    path: multilingual/c4-ig.*.json.gz700  - split: validation701    path: multilingual/c4-ig-validation.*.json.gz702- config_name: is703  data_files:704  - split: train705    path: multilingual/c4-is.*.json.gz706  - split: validation707    path: multilingual/c4-is-validation.*.json.gz708- config_name: it709  data_files:710  - split: train711    path: multilingual/c4-it.*.json.gz712  - split: validation713    path: multilingual/c4-it-validation.*.json.gz714- config_name: iw715  data_files:716  - split: train717    path: multilingual/c4-iw.*.json.gz718  - split: validation719    path: multilingual/c4-iw-validation.*.json.gz720- config_name: ja721  data_files:722  - split: train723    path: multilingual/c4-ja.*.json.gz724  - split: validation725    path: multilingual/c4-ja-validation.*.json.gz726- config_name: ja-Latn727  data_files:728  - split: train729    path: multilingual/c4-ja-Latn.*.json.gz730  - split: validation731    path: multilingual/c4-ja-Latn-validation.*.json.gz732- config_name: jv733  data_files:734  - split: train735    path: multilingual/c4-jv.*.json.gz736  - split: validation737    path: multilingual/c4-jv-validation.*.json.gz738- config_name: ka739  data_files:740  - split: train741    path: multilingual/c4-ka.*.json.gz742  - split: validation743    path: multilingual/c4-ka-validation.*.json.gz744- config_name: kk745  data_files:746  - split: train747    path: multilingual/c4-kk.*.json.gz748  - split: validation749    path: multilingual/c4-kk-validation.*.json.gz750- config_name: km751  data_files:752  - split: train753    path: multilingual/c4-km.*.json.gz754  - split: validation755    path: multilingual/c4-km-validation.*.json.gz756- config_name: kn757  data_files:758  - split: train759    path: multilingual/c4-kn.*.json.gz760  - split: validation761    path: multilingual/c4-kn-validation.*.json.gz762- config_name: ko763  data_files:764  - split: train765    path: multilingual/c4-ko.*.json.gz766  - split: validation767    path: multilingual/c4-ko-validation.*.json.gz768- config_name: ku769  data_files:770  - split: train771    path: multilingual/c4-ku.*.json.gz772  - split: validation773    path: multilingual/c4-ku-validation.*.json.gz774- config_name: ky775  data_files:776  - split: train777    path: multilingual/c4-ky.*.json.gz778  - split: validation779    path: multilingual/c4-ky-validation.*.json.gz780- config_name: la781  data_files:782  - split: train783    path: multilingual/c4-la.*.json.gz784  - split: validation785    path: multilingual/c4-la-validation.*.json.gz786- config_name: lb787  data_files:788  - split: train789    path: multilingual/c4-lb.*.json.gz790  - split: validation791    path: multilingual/c4-lb-validation.*.json.gz792- config_name: lo793  data_files:794  - split: train795    path: multilingual/c4-lo.*.json.gz796  - split: validation797    path: multilingual/c4-lo-validation.*.json.gz798- config_name: lt799  data_files:800  - split: train801    path: multilingual/c4-lt.*.json.gz802  - split: validation803    path: multilingual/c4-lt-validation.*.json.gz804- config_name: lv805  data_files:806  - split: train807    path: multilingual/c4-lv.*.json.gz808  - split: validation809    path: multilingual/c4-lv-validation.*.json.gz810- config_name: mg811  data_files:812  - split: train813    path: multilingual/c4-mg.*.json.gz814  - split: validation815    path: multilingual/c4-mg-validation.*.json.gz816- config_name: mi817  data_files:818  - split: train819    path: multilingual/c4-mi.*.json.gz820  - split: validation821    path: multilingual/c4-mi-validation.*.json.gz822- config_name: mk823  data_files:824  - split: train825    path: multilingual/c4-mk.*.json.gz826  - split: validation827    path: multilingual/c4-mk-validation.*.json.gz828- config_name: ml829  data_files:830  - split: train831    path: multilingual/c4-ml.*.json.gz832  - split: validation833    path: multilingual/c4-ml-validation.*.json.gz834- config_name: mn835  data_files:836  - split: train837    path: multilingual/c4-mn.*.json.gz838  - split: validation839    path: multilingual/c4-mn-validation.*.json.gz840- config_name: mr841  data_files:842  - split: train843    path: multilingual/c4-mr.*.json.gz844  - split: validation845    path: multilingual/c4-mr-validation.*.json.gz846- config_name: ms847  data_files:848  - split: train849    path: multilingual/c4-ms.*.json.gz850  - split: validation851    path: multilingual/c4-ms-validation.*.json.gz852- config_name: mt853  data_files:854  - split: train855    path: multilingual/c4-mt.*.json.gz856  - split: validation857    path: multilingual/c4-mt-validation.*.json.gz858- config_name: my859  data_files:860  - split: train861    path: multilingual/c4-my.*.json.gz862  - split: validation863    path: multilingual/c4-my-validation.*.json.gz864- config_name: ne865  data_files:866  - split: train867    path: multilingual/c4-ne.*.json.gz868  - split: validation869    path: multilingual/c4-ne-validation.*.json.gz870- config_name: nl871  data_files:872  - split: train873    path: multilingual/c4-nl.*.json.gz874  - split: validation875    path: multilingual/c4-nl-validation.*.json.gz876- config_name: 'no'877  data_files:878  - split: train879    path: multilingual/c4-no.*.json.gz880  - split: validation881    path: multilingual/c4-no-validation.*.json.gz882- config_name: ny883  data_files:884  - split: train885    path: multilingual/c4-ny.*.json.gz886  - split: validation887    path: multilingual/c4-ny-validation.*.json.gz888- config_name: pa889  data_files:890  - split: train891    path: multilingual/c4-pa.*.json.gz892  - split: validation893    path: multilingual/c4-pa-validation.*.json.gz894- config_name: pl895  data_files:896  - split: train897    path: multilingual/c4-pl.*.json.gz898  - split: validation899    path: multilingual/c4-pl-validation.*.json.gz900- config_name: ps901  data_files:902  - split: train903    path: multilingual/c4-ps.*.json.gz904  - split: validation905    path: multilingual/c4-ps-validation.*.json.gz906- config_name: pt907  data_files:908  - split: train909    path: multilingual/c4-pt.*.json.gz910  - split: validation911    path: multilingual/c4-pt-validation.*.json.gz912- config_name: ro913  data_files:914  - split: train915    path: multilingual/c4-ro.*.json.gz916  - split: validation917    path: multilingual/c4-ro-validation.*.json.gz918- config_name: ru919  data_files:920  - split: train921    path: multilingual/c4-ru.*.json.gz922  - split: validation923    path: multilingual/c4-ru-validation.*.json.gz924- config_name: ru-Latn925  data_files:926  - split: train927    path: multilingual/c4-ru-Latn.*.json.gz928  - split: validation929    path: multilingual/c4-ru-Latn-validation.*.json.gz930- config_name: sd931  data_files:932  - split: train933    path: multilingual/c4-sd.*.json.gz934  - split: validation935    path: multilingual/c4-sd-validation.*.json.gz936- config_name: si937  data_files:938  - split: train939    path: multilingual/c4-si.*.json.gz940  - split: validation941    path: multilingual/c4-si-validation.*.json.gz942- config_name: sk943  data_files:944  - split: train945    path: multilingual/c4-sk.*.json.gz946  - split: validation947    path: multilingual/c4-sk-validation.*.json.gz948- config_name: sl949  data_files:950  - split: train951    path: multilingual/c4-sl.*.json.gz952  - split: validation953    path: multilingual/c4-sl-validation.*.json.gz954- config_name: sm955  data_files:956  - split: train957    path: multilingual/c4-sm.*.json.gz958  - split: validation959    path: multilingual/c4-sm-validation.*.json.gz960- config_name: sn961  data_files:962  - split: train963    path: multilingual/c4-sn.*.json.gz964  - split: validation965    path: multilingual/c4-sn-validation.*.json.gz966- config_name: so967  data_files:968  - split: train969    path: multilingual/c4-so.*.json.gz970  - split: validation971    path: multilingual/c4-so-validation.*.json.gz972- config_name: sq973  data_files:974  - split: train975    path: multilingual/c4-sq.*.json.gz976  - split: validation977    path: multilingual/c4-sq-validation.*.json.gz978- config_name: sr979  data_files:980  - split: train981    path: multilingual/c4-sr.*.json.gz982  - split: validation983    path: multilingual/c4-sr-validation.*.json.gz984- config_name: st985  data_files:986  - split: train987    path: multilingual/c4-st.*.json.gz988  - split: validation989    path: multilingual/c4-st-validation.*.json.gz990- config_name: su991  data_files:992  - split: train993    path: multilingual/c4-su.*.json.gz994  - split: validation995    path: multilingual/c4-su-validation.*.json.gz996- config_name: sv997  data_files:998  - split: train999    path: multilingual/c4-sv.*.json.gz1000  - split: validation1001    path: multilingual/c4-sv-validation.*.json.gz1002- config_name: sw1003  data_files:1004  - split: train1005    path: multilingual/c4-sw.*.json.gz1006  - split: validation1007    path: multilingual/c4-sw-validation.*.json.gz1008- config_name: ta1009  data_files:1010  - split: train1011    path: multilingual/c4-ta.*.json.gz1012  - split: validation1013    path: multilingual/c4-ta-validation.*.json.gz1014- config_name: te1015  data_files:1016  - split: train1017    path: multilingual/c4-te.*.json.gz1018  - split: validation1019    path: multilingual/c4-te-validation.*.json.gz1020- config_name: tg1021  data_files:1022  - split: train1023    path: multilingual/c4-tg.*.json.gz1024  - split: validation1025    path: multilingual/c4-tg-validation.*.json.gz1026- config_name: th1027  data_files:1028  - split: train1029    path: multilingual/c4-th.*.json.gz1030  - split: validation1031    path: multilingual/c4-th-validation.*.json.gz1032- config_name: tr1033  data_files:1034  - split: train1035    path: multilingual/c4-tr.*.json.gz1036  - split: validation1037    path: multilingual/c4-tr-validation.*.json.gz1038- config_name: uk1039  data_files:1040  - split: train1041    path: multilingual/c4-uk.*.json.gz1042  - split: validation1043    path: multilingual/c4-uk-validation.*.json.gz1044- config_name: und1045  data_files:1046  - split: train1047    path: multilingual/c4-und.*.json.gz1048  - split: validation1049    path: multilingual/c4-und-validation.*.json.gz1050- config_name: ur1051  data_files:1052  - split: train1053    path: multilingual/c4-ur.*.json.gz1054  - split: validation1055    path: multilingual/c4-ur-validation.*.json.gz1056- config_name: uz1057  data_files:1058  - split: train1059    path: multilingual/c4-uz.*.json.gz1060  - split: validation1061    path: multilingual/c4-uz-validation.*.json.gz1062- config_name: vi1063  data_files:1064  - split: train1065    path: multilingual/c4-vi.*.json.gz1066  - split: validation1067    path: multilingual/c4-vi-validation.*.json.gz1068- config_name: xh1069  data_files:1070  - split: train1071    path: multilingual/c4-xh.*.json.gz1072  - split: validation1073    path: multilingual/c4-xh-validation.*.json.gz1074- config_name: yi1075  data_files:1076  - split: train1077    path: multilingual/c4-yi.*.json.gz1078  - split: validation1079    path: multilingual/c4-yi-validation.*.json.gz1080- config_name: yo1081  data_files:1082  - split: train1083    path: multilingual/c4-yo.*.json.gz1084  - split: validation1085    path: multilingual/c4-yo-validation.*.json.gz1086- config_name: zh1087  data_files:1088  - split: train1089    path: multilingual/c4-zh.*.json.gz1090  - split: validation1091    path: multilingual/c4-zh-validation.*.json.gz1092- config_name: zh-Latn1093  data_files:1094  - split: train1095    path: multilingual/c4-zh-Latn.*.json.gz1096  - split: validation1097    path: multilingual/c4-zh-Latn-validation.*.json.gz1098- config_name: zu1099  data_files:1100  - split: train1101    path: multilingual/c4-zu.*.json.gz1102  - split: validation1103    path: multilingual/c4-zu-validation.*.json.gz1104---1105 1106# C41107 1108## Dataset Description1109 1110- **Paper:** https://arxiv.org/abs/1910.106831111 1112### Dataset Summary1113 1114A colossal, cleaned version of Common Crawl's web crawl corpus. Based on Common Crawl dataset: "https://commoncrawl.org".1115 1116This is the processed version of [Google's C4 dataset](https://www.tensorflow.org/datasets/catalog/c4)1117 1118We prepared five variants of the data: `en`, `en.noclean`, `en.noblocklist`, `realnewslike`, and `multilingual` (mC4).1119 1120For reference, these are the sizes of the variants:1121 1122- `en`: 305GB1123- `en.noclean`: 2.3TB1124- `en.noblocklist`: 380GB1125- `realnewslike`: 15GB1126- `multilingual` (mC4): 9.7TB (108 subsets, one per language)1127 1128The `en.noblocklist` variant is exactly the same as the `en` variant, except we turned off the so-called "badwords filter", which removes all documents that contain words from the lists at https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words.1129 1130#### How do I download this?1131 1132##### Using 🤗 Datasets1133 1134```python1135from datasets import load_dataset1136 1137# English only1138en = load_dataset("allenai/c4", "en")1139 1140# Other variants in english1141en_noclean = load_dataset("allenai/c4", "en.noclean")1142en_noblocklist = load_dataset("allenai/c4", "en.noblocklist")1143realnewslike = load_dataset("allenai/c4", "realnewslike")1144 1145# Multilingual (108 languages)1146multilingual = load_dataset("allenai/c4", "multilingual")1147 1148# One specific language1149es = load_dataset("allenai/c4", "es")1150```1151 1152Since this dataset is big, it is encouraged to load it in streaming mode using `streaming=True`, for example:1153 1154```python1155en = load_dataset("allenai/c4", "en", streaming=True)1156```1157 1158You can also load and mix multiple languages:1159 1160```python1161from datasets import concatenate_datasets, interleave_datasets, load_dataset1162 1163es = load_dataset("allenai/c4", "es", streaming=True)1164fr = load_dataset("allenai/c4", "fr", streaming=True)1165 1166# Concatenate both datasets1167concatenated = concatenate_datasets([es, fr])1168# Or interleave them (alternates between one and the other)1169interleaved = interleave_datasets([es, fr])1170```1171 1172##### Using Dask1173 1174```python1175import dask.dataframe as dd1176 1177df = dd.read_json("hf://datasets/allenai/c4/en/c4-train.*.json.gz")1178 1179# English only1180en_df = dd.read_json("hf://datasets/allenai/c4/en/c4-*.json.gz")1181 1182# Other variants in english1183en_noclean_df = dd.read_json("hf://datasets/allenai/c4/en/noclean/c4-*.json.gz")1184en_noblocklist_df = dd.read_json("hf://datasets/allenai/c4/en.noblocklist/c4-*.json.gz")1185realnewslike_df = dd.read_json("hf://datasets/allenai/c4/realnewslike/c4-*.json.gz")1186 1187# Multilingual (108 languages)1188multilingual_df = dd.read_json("hf://datasets/allenai/c4/multilingual/c4-*.json.gz")1189 1190# One specific language1191es_train_df = dd.read_json("hf://datasets/allenai/c4/multilingual/c4-es.*.json.gz")1192es_valid_df = dd.read_json("hf://datasets/allenai/c4/multilingual/c4-es-validation.*.json.gz")1193```1194 1195##### Using Git1196 1197```bash1198git clone https://huggingface.co/datasets/allenai/c41199```1200 

Showing the first 1,200 of 1391 lines. Download the file for the rest.