BEE-spoke-data/awesome-python-apps
Dataset Card for "awesome-python-apps" This contains .py files for the following repos taken from awesome-python-applications (on GitHub here) abilian-sbe clone_repos.sh invesalius3 photonix sk1-wx ambar CONTRIBUTING.md isso picard soundconverter apatite CTFd kibitzr pi-hole soundgrain ArchiveBox Cura KindleEar planet… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/awesome-python-apps.
177
1---2language:3- en4license: odc-by5size_categories:6- 10K<n<100K7task_categories:8- text-generation9configs:10- config_name: all11 data_files:12 - split: train13 path: all/train-*14 - split: validation15 path: all/validation-*16 - split: test17 path: all/test-*18- config_name: c_and_cpp19 data_files:20 - split: train21 path: c_and_cpp/train-*22 - split: validation23 path: c_and_cpp/validation-*24 - split: test25 path: c_and_cpp/test-*26- config_name: default27 data_files:28 - split: train29 path: data/train-*30 - split: validation31 path: data/validation-*32 - split: test33 path: data/test-*34- config_name: docs_and_configs35 data_files:36 - split: train37 path: docs_and_configs/train-*38 - split: validation39 path: docs_and_configs/validation-*40 - split: test41 path: docs_and_configs/test-*42- config_name: javascript43 data_files:44 - split: train45 path: javascript/train-*46 - split: validation47 path: javascript/validation-*48 - split: test49 path: javascript/test-*50dataset_info:51- config_name: all52 features:53 - name: section54 dtype: string55 - name: filename56 dtype: string57 - name: text58 dtype: string59 splits:60 - name: train61 num_bytes: 49875109362 num_examples: 2749863 - name: validation64 num_bytes: 2663828365 num_examples: 152766 - name: test67 num_bytes: 2599945468 num_examples: 153069 download_size: 18155923070 dataset_size: 55138883071- config_name: c_and_cpp72 features:73 - name: section74 dtype: string75 - name: filename76 dtype: string77 - name: text78 dtype: string79 splits:80 - name: train81 num_bytes: 77766288.4875330982 num_examples: 815783 - name: validation84 num_bytes: 4318760.41251103385 num_examples: 45386 - name: test87 num_bytes: 4328294.0999558788 num_examples: 45489 download_size: 2992061290 dataset_size: 8641334391- config_name: default92 features:93 - name: section94 dtype: string95 - name: filename96 dtype: string97 - name: text98 dtype: string99 splits:100 - name: train101 num_bytes: 187113233.13447574102 num_examples: 12394103 - name: validation104 num_bytes: 10401889.432762126105 num_examples: 689106 - name: test107 num_bytes: 10401889.432762126108 num_examples: 689109 download_size: 74676703110 dataset_size: 207917011.99999997111- config_name: docs_and_configs112 features:113 - name: section114 dtype: string115 - name: filename116 dtype: string117 - name: text118 dtype: string119 splits:120 - name: train121 num_bytes: 27004737.076526225122 num_examples: 3140123 - name: validation124 num_bytes: 1496440.8443680138125 num_examples: 174126 - name: test127 num_bytes: 1505041.079105761128 num_examples: 175129 download_size: 14090699130 dataset_size: 30006219131- config_name: javascript132 features:133 - name: section134 dtype: string135 - name: filename136 dtype: string137 - name: text138 dtype: string139 splits:140 - name: train141 num_bytes: 204347030.4142 num_examples: 3807143 - name: validation144 num_bytes: 11325774.471867612145 num_examples: 211146 - name: test147 num_bytes: 11379451.128132388148 num_examples: 212149 download_size: 60838679150 dataset_size: 227052256151tags:152- code153- python154- cpp155- javascript156---157# Dataset Card for "awesome-python-apps"158 159 160This contains `.py` files for the following repos taken from `awesome-python-applications` (on GitHub [here](https://github.com/mahmoud/awesome-python-applications))161 162 163<pre><font color="#268BD2"><b>abilian-sbe</b></font> <font color="#859900"><b>clone_repos.sh</b></font> <font color="#268BD2"><b>invesalius3</b></font> <font color="#268BD2"><b>photonix</b></font> <font color="#268BD2"><b>sk1-wx</b></font>164<font color="#268BD2"><b>ambar</b></font> <font color="#859900"><b>CONTRIBUTING.md</b></font> <font color="#268BD2"><b>isso</b></font> <font color="#268BD2"><b>picard</b></font> <font color="#268BD2"><b>soundconverter</b></font>165<font color="#268BD2"><b>apatite</b></font> <font color="#268BD2"><b>CTFd</b></font> <font color="#268BD2"><b>kibitzr</b></font> <font color="#268BD2"><b>pi-hole</b></font> <font color="#268BD2"><b>soundgrain</b></font>166<font color="#268BD2"><b>ArchiveBox</b></font> <font color="#268BD2"><b>Cura</b></font> <font color="#268BD2"><b>KindleEar</b></font> <font color="#268BD2"><b>planet</b></font> <font color="#268BD2"><b>stargate</b></font>167<font color="#268BD2"><b>archivematica</b></font> <font color="#268BD2"><b>deluge</b></font> <font color="#268BD2"><b>Lector</b></font> <font color="#268BD2"><b>plover</b></font> <font color="#268BD2"><b>streamlink</b></font>168<font color="#268BD2"><b>autokey</b></font> <font color="#268BD2"><b>dissemin</b></font> <font color="#268BD2"><b>lucaschess</b></font> <font color="#268BD2"><b>pol</b></font> <font color="#268BD2"><b>Sunflower</b></font>169<font color="#268BD2"><b>awesome-python-applications</b></font> <font color="#268BD2"><b>drawbot</b></font> <font color="#268BD2"><b>mackup</b></font> <font color="#268BD2"><b>posthog</b></font> <font color="#268BD2"><b>supysonic</b></font>170<font color="#268BD2"><b>babybuddy</b></font> <font color="#268BD2"><b>exaile</b></font> <font color="#268BD2"><b>meshroom</b></font> <font color="#859900"><b>projects.yaml</b></font> <font color="#268BD2"><b>syncserver</b></font>171<font color="#268BD2"><b>beancount</b></font> <font color="#268BD2"><b>flaskbb</b></font> <font color="#268BD2"><b>mopidy</b></font> <font color="#268BD2"><b>puddletag</b></font> <font color="#268BD2"><b>templates</b></font>172<font color="#268BD2"><b>beets</b></font> <font color="#268BD2"><b>flowblade</b></font> <font color="#268BD2"><b>music-player</b></font> <font color="#268BD2"><b>PyBitmessage</b></font> <font color="#268BD2"><b>term2048</b></font>173<font color="#268BD2"><b>bleachbit</b></font> <font color="#268BD2"><b>fofix</b></font> <font color="#268BD2"><b>mylar</b></font> <font color="#268BD2"><b>Pyfa</b></font> <font color="#268BD2"><b>thumbor</b></font>174<font color="#268BD2"><b>bookwyrm</b></font> <font color="#268BD2"><b>formspree</b></font> <font color="#268BD2"><b>mypaint</b></font> <font color="#268BD2"><b>pyload</b></font> <font color="#859900"><b>TODO.md</b></font>175<font color="#268BD2"><b>borg</b></font> <font color="#268BD2"><b>FreeCAD</b></font> <font color="#268BD2"><b>neubot</b></font> <font color="#268BD2"><b>pynocchio</b></font> <font color="#268BD2"><b>tribler</b></font>176<font color="#268BD2"><b>buku</b></font> <font color="#268BD2"><b>frescobaldi</b></font> <font color="#268BD2"><b>newspipe</b></font> <font color="#268BD2"><b>pyvideo</b></font> <font color="#268BD2"><b>vidcutter</b></font>177<font color="#268BD2"><b>Bup</b></font> <font color="#268BD2"><b>friture</b></font> <font color="#268BD2"><b>nfoview</b></font> <font color="#268BD2"><b>qis</b></font> <font color="#268BD2"><b>whipper</b></font>178<font color="#859900"><b>BY_PLATFORM.md</b></font> <font color="#268BD2"><b>gaphor</b></font> <font color="#268BD2"><b>notebooks</b></font> <font color="#268BD2"><b>quodlibet</b></font> <font color="#268BD2"><b>you-get</b></font>179<font color="#268BD2"><b>Cactus</b></font> <font color="#268BD2"><b>-github</b></font> <font color="#268BD2"><b>nyaa</b></font> <font color="#268BD2"><b>qutebrowser</b></font> <font color="#268BD2"><b>youtube-dl</b></font>180<font color="#268BD2"><b>canto-curses</b></font> <font color="#268BD2"><b>gnuradio</b></font> <font color="#268BD2"><b>ocropy</b></font> <font color="#859900"><b>README.md</b></font> <font color="#268BD2"><b>ZeroNet</b></font>181<font color="#268BD2"><b>canto-next</b></font> <font color="#268BD2"><b>gpodder</b></font> <font color="#268BD2"><b>OctoPrint</b></font> <font color="#268BD2"><b>reddit</b></font>182<font color="#268BD2"><b>cartoonify</b></font> <font color="#268BD2"><b>hangups</b></font> <font color="#268BD2"><b>oncall</b></font> <font color="#859900"><b>revisit.yaml</b></font>183<font color="#268BD2"><b>CDDA-Game-Launcher</b></font> <font color="#268BD2"><b>hosts</b></font> <font color="#268BD2"><b>openshot-qt</b></font> <font color="#268BD2"><b>sabnzbd</b></font>184<font color="#268BD2"><b>ckan</b></font> <font color="#268BD2"><b>httpie</b></font> <font color="#268BD2"><b>PhotoCollage</b></font> <font color="#268BD2"><b>searx</b></font>185</pre>186 187 188## details189 190There are different configs for different languages and one for 'dcoumentation' type stuff. 191 192All of them should have the same column names, so if you want to use all of them for training it should be easy to aggregate them, just use the `concatenate_datasets` function + shuffle193 194### by config195 196#### python (default)197 198- all of them have been formatted with `ruff`199 200counts:201 202<pre>(ki) <font color="#859900"><b>➜ </b></font><font color="#2AA198"><b>primerdata-for-LLMs</b></font> python203Python 3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0] on linux204Type "help", "copyright", "credits" or "license" for more information.205>>> from datasets import load_dataset206>>> 207>>> # If the dataset is gated/private, make sure you have run huggingface-cli login208>>> dataset = load_dataset("BEE-spoke-data/awesome-python-apps")209>>> dataset210DatasetDict({211 train: Dataset({212 features: ['section', 'filename', 'text'],213 num_rows: 12394214 })215 validation: Dataset({216 features: ['section', 'filename', 'text'],217 num_rows: 689218 })219 test: Dataset({220 features: ['section', 'filename', 'text'],221 num_rows: 689222 })223})224</pre>225 226 227##### token counts228 229 230 231Llama (`use_fast=False`):232 233<pre> token_len234count 12394.0235mean 7863.521704236std 266330.944961237min 254.023825% 727.023950% 1417.024075% 3083.5241max 27741612.0242</pre>243 244NeoX:245 246<pre> token_len247count 12394.0248mean 6615.093432249std 213090.741642250min 234.025125% 677.025250% 1326.025375% 2883.75254max 22081563.0255</pre>256 257#### javascript258 259<pre>260INFO:__main__:Looking for files with extensions: ['js', 'ts', 'tsx']261Processing js files: 100%|██████████████████████████| 1441/1441 [00:00<00:00, 2232.28it/s]262Processing ts files: 100%|██████████████████████████| 1717/1717 [00:01<00:00, 1343.75it/s]263Processing tsx files: 100%|█████████████████████████| 1072/1072 [00:00<00:00, 3929.08it/s]264INFO:__main__:Found 4230 text files.265INFO:__main__:Performing train-test split...266INFO:__main__:Performing validation-test split...267INFO:__main__:Train size: 3807268INFO:__main__:Validation size: 211269INFO:__main__:Test size: 212270INFO:__main__:Pushing dataset to Huggingface hub (BEE-spoke-data/awesome-python-apps)...271INFO:__main__:Using repo id: BEE-spoke-data/awesome-python-apps, config name: javascript272</pre>273 274#### c_and_cpp275 276<pre>277INFO:__main__:Looking for files with extensions: ['c', 'h', 'i', 'cc', 'cpp', 'cxx', 'hpp', 'hxx', 'inl', 'ino', 'ixx', 'jxx', 'lxx', 'yxx']278Processing c files: 100%|█████████████████████████████| 450/450 [00:00<00:00, 2741.92it/s]279Processing h files: 100%|███████████████████████████| 3616/3616 [00:00<00:00, 4250.46it/s]280Processing i files: 100%|██████████████████████████████████| 1/1 [00:00<00:00, 197.47it/s]281Processing cc files: 100%|██████████████████████████| 1386/1386 [00:00<00:00, 4209.33it/s]282Processing cpp files: 100%|█████████████████████████| 3015/3015 [00:00<00:00, 3254.90it/s]283Processing cxx files: 100%|█████████████████████████████| 10/10 [00:00<00:00, 1203.77it/s]284Processing hpp files: 100%|███████████████████████████| 268/268 [00:00<00:00, 3523.76it/s]285Processing hxx files: 100%|███████████████████████████| 257/257 [00:00<00:00, 3406.82it/s]286Processing inl files: 100%|█████████████████████████████| 56/56 [00:00<00:00, 2770.02it/s]287Processing ino files: 100%|████████████████████████████████| 1/1 [00:00<00:00, 195.83it/s]288Processing ixx files: 100%|████████████████████████████████| 1/1 [00:00<00:00, 200.43it/s]289Processing jxx files: 100%|████████████████████████████████| 1/1 [00:00<00:00, 187.72it/s]290Processing lxx files: 100%|████████████████████████████████| 1/1 [00:00<00:00, 185.00it/s]291Processing yxx files: 100%|████████████████████████████████| 1/1 [00:00<00:00, 192.69it/s]292INFO:__main__:Found 9064 text files.293INFO:__main__:Performing train-test split...294INFO:__main__:Performing validation-test split...295INFO:__main__:Train size: 8157296INFO:__main__:Validation size: 453297INFO:__main__:Test size: 454298INFO:__main__:Pushing dataset to Huggingface hub (BEE-spoke-data/awesome-python-apps)...299INFO:__main__:Using repo id: BEE-spoke-data/awesome-python-apps, config name: c_and_cpp300</pre>301 302 303 304#### docs and configs305 306<pre>INFO:__main__:Looking for files with extensions: ['md', 'mdx', 'rst', 'txt', 'env', 'yml', 'yaml', 'toml', 'cfg', 'conf', 'config', 'gitignore', 'MD', 'mkd']307Processing md files: 100%|████████████████████████████| 741/741 [00:00<00:00, 3664.75it/s]308Processing mdx files: 100%|███████████████████████████████| 8/8 [00:00<00:00, 1168.29it/s]309Processing rst files: 100%|███████████████████████████| 699/699 [00:00<00:00, 3901.65it/s]310Processing txt files: 100%|███████████████████████████| 646/646 [00:00<00:00, 3348.51it/s]311Processing env files: 100%|████████████████████████████████| 1/1 [00:00<00:00, 188.49it/s]312Processing yml files: 100%|███████████████████████████| 650/650 [00:00<00:00, 4145.67it/s]313Processing yaml files: 100%|████████████████████████████| 62/62 [00:00<00:00, 2146.06it/s]314Processing toml files: 100%|████████████████████████████| 13/13 [00:00<00:00, 1934.71it/s]315Processing cfg files: 100%|███████████████████████████| 618/618 [00:00<00:00, 3866.66it/s]316Processing conf files: 100%|████████████████████████████| 21/21 [00:00<00:00, 2169.95it/s]317Processing config files: 100%|██████████████████████████| 20/20 [00:00<00:00, 2146.14it/s]318Processing gitignore files: 100%|██████████████████████████| 6/6 [00:00<00:00, 890.83it/s]319Processing MD files: 100%|█████████████████████████████████| 1/1 [00:00<00:00, 197.70it/s]320Processing mkd files: 100%|███████████████████████████████| 3/3 [00:00<00:00, 3669.56it/s]321INFO:__main__:Found 3489 text files.322INFO:__main__:Performing train-test split...323INFO:__main__:Performing validation-test split...324INFO:__main__:Train size: 3140325INFO:__main__:Validation size: 174326INFO:__main__:Test size: 175327INFO:__main__:Pushing dataset to Huggingface hub (BEE-spoke-data/awesome-python-apps)...328INFO:__main__:Using repo id: BEE-spoke-data/awesome-python-apps, config name: docs_and_configs329</pre>