Datasetpublic

BEE-spoke-data/awesome-python-apps

Dataset Card for "awesome-python-apps" This contains .py files for the following repos taken from awesome-python-applications (on GitHub here) abilian-sbe clone_repos.sh invesalius3 photonix sk1-wx ambar CONTRIBUTING.md isso picard soundconverter apatite CTFd kibitzr pi-hole soundgrain ArchiveBox Cura KindleEar planet… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/awesome-python-apps.

sourceHugging Faceodc-byupdated 6mo agoView on Hugging Face

1likes77downloads

Dataset Card

Dataset Card for "awesome-python-apps"

This contains .py files for the following repos taken from awesome-python-applications (on GitHub here)

<pre>abilian-sbe clonerepos.sh invesalius3 photonix sk1-wx ambar CONTRIBUTING.md isso picard soundconverter apatite CTFd kibitzr pi-hole soundgrain ArchiveBox Cura KindleEar planet stargate archivematica deluge Lector plover streamlink autokey dissemin lucaschess pol Sunflower awesome-python-applications drawbot mackup posthog supysonic babybuddy exaile meshroom projects.yaml syncserver beancount flaskbb mopidy puddletag templates beets flowblade music-player PyBitmessage term2048 bleachbit fofix mylar Pyfa thumbor bookwyrm formspree mypaint pyload TODO.md borg FreeCAD neubot pynocchio tribler buku frescobaldi newspipe pyvideo vidcutter Bup friture nfoview qis whipper BYPLATFORM.md gaphor notebooks quodlibet you-get Cactus -github nyaa qutebrowser youtube-dl canto-curses gnuradio ocropy README.md ZeroNet canto-next gpodder OctoPrint reddit cartoonify hangups oncall revisit.yaml CDDA-Game-Launcher hosts openshot-qt sabnzbd ckan httpie PhotoCollage searx </pre>

details

There are different configs for different languages and one for 'dcoumentation' type stuff.

All of them should have the same column names, so if you want to use all of them for training it should be easy to aggregate them, just use the concatenate_datasets function + shuffle

by config

python (default)

—all of them have been formatted with ruff

counts:

<pre>(ki) ➜ primerdata-for-LLMs python Python 3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> from datasets import loaddataset >>> >>> # If the dataset is gated/private, make sure you have run huggingface-cli login >>> dataset = loaddataset("BEE-spoke-data/awesome-python-apps") >>> dataset DatasetDict({ train: Dataset({ features: ['section', 'filename', 'text'], numrows: 12394 }) validation: Dataset({ features: ['section', 'filename', 'text'], numrows: 689 }) test: Dataset({ features: ['section', 'filename', 'text'], num_rows: 689 }) }) </pre>

token counts

Llama (use_fast=False):

<pre> token_len count 12394.0 mean 7863.521704 std 266330.944961 min 254.0 25% 727.0 50% 1417.0 75% 3083.5 max 27741612.0 </pre>

NeoX:

<pre> token_len count 12394.0 mean 6615.093432 std 213090.741642 min 234.0 25% 677.0 50% 1326.0 75% 2883.75 max 22081563.0 </pre>

javascript

<pre> INFO:_main:Looking for files with extensions: ['js', 'ts', 'tsx'] Processing js files: 100%|██████████████████████████| 1441/1441 [00:00<00:00, 2232.28it/s] Processing ts files: 100%|██████████████████████████| 1717/1717 [00:01<00:00, 1343.75it/s] Processing tsx files: 100%|█████████████████████████| 1072/1072 [00:00<00:00, 3929.08it/s] INFO:main:Found 4230 text files. INFO:main:Performing train-test split... INFO:main:Performing validation-test split... INFO:main:Train size: 3807 INFO:main:Validation size: 211 INFO:main:Test size: 212 INFO:main:Pushing dataset to Huggingface hub (BEE-spoke-data/awesome-python-apps)... INFO:main_:Using repo id: BEE-spoke-data/awesome-python-apps, config name: javascript </pre>

candcpp

<pre> INFO:_main:Looking for files with extensions: ['c', 'h', 'i', 'cc', 'cpp', 'cxx', 'hpp', 'hxx', 'inl', 'ino', 'ixx', 'jxx', 'lxx', 'yxx'] Processing c files: 100%|█████████████████████████████| 450/450 [00:00<00:00, 2741.92it/s] Processing h files: 100%|███████████████████████████| 3616/3616 [00:00<00:00, 4250.46it/s] Processing i files: 100%|██████████████████████████████████| 1/1 [00:00<00:00, 197.47it/s] Processing cc files: 100%|██████████████████████████| 1386/1386 [00:00<00:00, 4209.33it/s] Processing cpp files: 100%|█████████████████████████| 3015/3015 [00:00<00:00, 3254.90it/s] Processing cxx files: 100%|█████████████████████████████| 10/10 [00:00<00:00, 1203.77it/s] Processing hpp files: 100%|███████████████████████████| 268/268 [00:00<00:00, 3523.76it/s] Processing hxx files: 100%|███████████████████████████| 257/257 [00:00<00:00, 3406.82it/s] Processing inl files: 100%|█████████████████████████████| 56/56 [00:00<00:00, 2770.02it/s] Processing ino files: 100%|████████████████████████████████| 1/1 [00:00<00:00, 195.83it/s] Processing ixx files: 100%|████████████████████████████████| 1/1 [00:00<00:00, 200.43it/s] Processing jxx files: 100%|████████████████████████████████| 1/1 [00:00<00:00, 187.72it/s] Processing lxx files: 100%|████████████████████████████████| 1/1 [00:00<00:00, 185.00it/s] Processing yxx files: 100%|████████████████████████████████| 1/1 [00:00<00:00, 192.69it/s] INFO:main:Found 9064 text files. INFO:main:Performing train-test split... INFO:main:Performing validation-test split... INFO:main:Train size: 8157 INFO:main:Validation size: 453 INFO:main:Test size: 454 INFO:main:Pushing dataset to Huggingface hub (BEE-spoke-data/awesome-python-apps)... INFO:main:Using repo id: BEE-spoke-data/awesome-python-apps, config name: cand_cpp </pre>

docs and configs

<pre>INFO:_main:Looking for files with extensions: ['md', 'mdx', 'rst', 'txt', 'env', 'yml', 'yaml', 'toml', 'cfg', 'conf', 'config', 'gitignore', 'MD', 'mkd'] Processing md files: 100%|████████████████████████████| 741/741 [00:00<00:00, 3664.75it/s] Processing mdx files: 100%|███████████████████████████████| 8/8 [00:00<00:00, 1168.29it/s] Processing rst files: 100%|███████████████████████████| 699/699 [00:00<00:00, 3901.65it/s] Processing txt files: 100%|███████████████████████████| 646/646 [00:00<00:00, 3348.51it/s] Processing env files: 100%|████████████████████████████████| 1/1 [00:00<00:00, 188.49it/s] Processing yml files: 100%|███████████████████████████| 650/650 [00:00<00:00, 4145.67it/s] Processing yaml files: 100%|████████████████████████████| 62/62 [00:00<00:00, 2146.06it/s] Processing toml files: 100%|████████████████████████████| 13/13 [00:00<00:00, 1934.71it/s] Processing cfg files: 100%|███████████████████████████| 618/618 [00:00<00:00, 3866.66it/s] Processing conf files: 100%|████████████████████████████| 21/21 [00:00<00:00, 2169.95it/s] Processing config files: 100%|██████████████████████████| 20/20 [00:00<00:00, 2146.14it/s] Processing gitignore files: 100%|██████████████████████████| 6/6 [00:00<00:00, 890.83it/s] Processing MD files: 100%|█████████████████████████████████| 1/1 [00:00<00:00, 197.70it/s] Processing mkd files: 100%|███████████████████████████████| 3/3 [00:00<00:00, 3669.56it/s] INFO:main:Found 3489 text files. INFO:main:Performing train-test split... INFO:main:Performing validation-test split... INFO:main:Train size: 3140 INFO:main:Validation size: 174 INFO:main:Test size: 175 INFO:main:Pushing dataset to Huggingface hub (BEE-spoke-data/awesome-python-apps)... INFO:main:Using repo id: BEE-spoke-data/awesome-python-apps, config name: docsand_configs </pre>