Datasetpublic

macrocosm-os/code-parrot-github-code

GitHub Code Dataset Dataset Description The GitHub Code dataset consists of 115M code files from GitHub in 32 programming languages with 60 extensions totaling in 1TB of data. The dataset was created from the public GitHub dataset on Google BiqQuery. How to use it The GitHub Code dataset is a very large dataset so for most use cases it is recommended to make use of the streaming API of datasets. You can load and iterate through the dataset with the… See the full description on the dataset page: https://huggingface.co/datasets/macrocosm-os/code-parrot-github-code.

sourceHugging Faceotherupdated 2y agoView on Hugging Face

13likes1.4kdownloads

rootquery.sql

query.sql84 linesDownload Raw Back to root

1SELECT2  f.repo_name,3  f.path,4  c.copies,5  c.size,6  c.content,7  l.license8FROM9 (select f.*, row_number() over (partition by id order by path desc) as seqnum10      from `bigquery-public-data.github_repos.files` AS f) f11JOIN12  `bigquery-public-data.github_repos.contents` AS c13ON14  f.id = c.id AND seqnum=115JOIN16  `bigquery-public-data.github_repos.licenses` AS l17ON18  f.repo_name = l.repo_name19WHERE20  NOT c.binary21  AND ((f.path LIKE '%.asm'22      OR f.path LIKE '%.bat'23      OR f.path LIKE '%.cmd'24      OR f.path LIKE '%.c'25      OR f.path LIKE '%.h'26      OR f.path LIKE '%.cs'27      OR f.path LIKE '%.cpp'28      OR f.path LIKE '%.hpp'29      OR f.path LIKE '%.c++'30      OR f.path LIKE '%.h++'31      OR f.path LIKE '%.cc'32      OR f.path LIKE '%.hh'33      OR f.path LIKE '%.C'34      OR f.path LIKE '%.H'35      OR f.path LIKE '%.cmake'36      OR f.path LIKE '%.css'37      OR f.path LIKE '%.dockerfile'38      OR f.path LIKE '%.f90'39      OR f.path LIKE '%.f'40      OR f.path LIKE '%.f03'41      OR f.path LIKE '%.f08'42      OR f.path LIKE '%.f77'43      OR f.path LIKE '%.f95'44      OR f.path LIKE '%.for'45      OR f.path LIKE '%.fpp'46      OR f.path LIKE '%.go'47      OR f.path LIKE '%.hs'48      OR f.path LIKE '%.html'49      OR f.path LIKE '%.java'50      OR f.path LIKE '%.js'51      OR f.path LIKE '%.jl'52      OR f.path LIKE '%.lua'53      OR f.path LIKE '%.md'54      OR f.path LIKE '%.markdown'55      OR f.path LIKE '%.php'56      OR f.path LIKE '%.php3'57      OR f.path LIKE '%.php4'58      OR f.path LIKE '%.php5'59      OR f.path LIKE '%.phps'60      OR f.path LIKE '%.phpt'61      OR f.path LIKE '%.pl'62      OR f.path LIKE '%.pm'63      OR f.path LIKE '%.pod'64      OR f.path LIKE '%.perl'65      OR f.path LIKE '%.ps1'66      OR f.path LIKE '%.psd1'67      OR f.path LIKE '%.psm1'68      OR f.path LIKE '%.py'69      OR f.path LIKE '%.rb'70      OR f.path LIKE '%.rs'71      OR f.path LIKE '%.sql'72      OR f.path LIKE '%.scala'73      OR f.path LIKE '%.sh'74      OR f.path LIKE '%.bash'75      OR f.path LIKE '%.command'76      OR f.path LIKE '%.zsh'77      OR f.path LIKE '%.ts'78      OR f.path LIKE '%.tsx'79      OR f.path LIKE '%.tex'80      OR f.path LIKE '%.vb'81      OR f.path LIKE '%Dockerfile'82      OR f.path LIKE '%Makefile')83    AND (c.size BETWEEN 084      AND 1048575))