Datasetpublic

codeparrot/github-code

The GitHub Code dataest consists of 115M code files from GitHub in 32 programming languages with 60 extensions totalling in 1TB of text data. The dataset was created from the GitHub dataset on BiqQuery.

sourceHugging Faceotherupdated 4y agoView on Hugging Face

420likes40kdownloads

rootquery.sql

query.sql81 linesDownload Raw Back to root

1SELECT2  f.repo_name,3  f.path,4  c.copies,5  c.size,6  c.content,7  l.license8FROM9 (select f.*, row_number() over (partition by id order by path desc) as seqnum10      from `bigquery-public-data.github_repos.files` AS f) f11JOIN12  `bigquery-public-data.github_repos.contents` AS c13ON14  f.id = c.id AND seqnum=115JOIN16  `bigquery-public-data.github_repos.licenses` AS l17ON18  f.repo_name = l.repo_name19WHERE20  NOT c.binary21  AND ((f.path LIKE '%.asm'22      OR f.path LIKE '%.bat'23      OR f.path LIKE '%.cmd'24      OR f.path LIKE '%.c'25      OR f.path LIKE '%.h'26      OR f.path LIKE '%.cs'27      OR f.path LIKE '%.cpp'28      OR f.path LIKE '%.hpp'29      OR f.path LIKE '%.c++'30      OR f.path LIKE '%.h++'31      OR f.path LIKE '%.cc'32      OR f.path LIKE '%.hh'33      OR f.path LIKE '%.C'34      OR f.path LIKE '%.H'35      OR f.path LIKE '%.cmake'36      OR f.path LIKE '%.css'37      OR f.path LIKE '%.dockerfile'38      OR f.path LIKE '%.f90'39      OR f.path LIKE '%.f'40      OR f.path LIKE '%.f03'41      OR f.path LIKE '%.f08'42      OR f.path LIKE '%.f77'43      OR f.path LIKE '%.f95'44      OR f.path LIKE '%.for'45      OR f.path LIKE '%.fpp'46      OR f.path LIKE '%.go'47      OR f.path LIKE '%.hs'48      OR f.path LIKE '%.html'49      OR f.path LIKE '%.java'50      OR f.path LIKE '%.js'51      OR f.path LIKE '%.jl'52      OR f.path LIKE '%.lua'53      OR f.path LIKE '%.md'54      OR f.path LIKE '%.markdown'55      OR f.path LIKE '%.php'56      OR f.path LIKE '%.php3'57      OR f.path LIKE '%.php4'58      OR f.path LIKE '%.php5'59      OR f.path LIKE '%.phps'60      OR f.path LIKE '%.phpt'61      OR f.path LIKE '%.pl'62      OR f.path LIKE '%.pm'63      OR f.path LIKE '%.pod'64      OR f.path LIKE '%.perl'65      OR f.path LIKE '%.ps1'66      OR f.path LIKE '%.psd1'67      OR f.path LIKE '%.psm1'68      OR f.path LIKE '%.py'69      OR f.path LIKE '%.rb'70      OR f.path LIKE '%.rs'71      OR f.path LIKE '%.sql'72      OR f.path LIKE '%.sh'73      OR f.path LIKE '%.bash'74      OR f.path LIKE '%.command'75      OR f.path LIKE '%.zsh'76      OR f.path LIKE '%.tex'77      OR f.path LIKE '%.vb'78      OR f.path LIKE '%Dockerfile'79      OR f.path LIKE '%Makefile')80    AND (c.size BETWEEN 081      AND 1048575))