bigcode/commits_sample_files

This is a sample of GitHub commits and files reconstructed from the Software Heritage dataset. It contains the latest 1024 commits in all of pytorch/* and huggingface/* repos.

The tables are split to avoid an explosion of rows (lots of repeated files between commits), so you will need to pre-filter the commits before adding the file contents.

Table descriptions:

1. commits

The commit message table. Join it with commit_filepath on commits.directory_id == commit_filepath.directory_id and commits.parent_directory_id == commit_filepath.directory_id.

• —origin - str (repo url, example: https://github.com/huggingface/datasets)
• —full_name - str (repo name, example: huggingface/datasets)
• —commit_id - str (example: 56b114ebfd5399252dc23f9df207f87c5397b50a)
• —parentcommitid - str (previous commit id, example: 8c826fb80f7f8135f6e632d34c8f59134f5983c8)
• —snapshot_id - str (Software Heritage snapshot id, example: d76232879a5912b1eaca91e8889863117bca66a4)
• —visit_date - datetime[ns] (Software Heritage crawler visit date, example: 2022-11-28 13:03:09.100114)
• —branch_name - str (example: refs/heads/main)
• —revision_date - datetime[ns] (commit date, example: 2022-04-27 17:30:41)
• —committer_date - datetime[ns] (commit date, example: 2022-04-27 17:30:41)
• —author - binary (Software Heritage anonymized commit's author name)
• —message - str (commit message, example: update auth when mirroring datasets on the hub (#4242))
• —directory_id - str (root directory id to join this table with the files, example: 84c6cc5b2c156ed3251674c43dd411d731183bb3)
• —parentdirectoryid - str (parent commit's root directory id, example: e927ce1cdecf6286f7e23204ed656373c9921f89)

2. commit_filepath

The file paths associated with each commit. Join it with file_conents on blob_id

• —directory_id - str (root directory id of the corresponding commit, example: 001331910958befd665d94c85c23471a8fc1ab19)
• —blob_id - str (Software Heritage file blob id, example: 47953673b7b51c2585402a91d434f5fe4d9dc105)
• —content_id - str (Software Heritage content id, example: d6b3dab547a59efe5246edf06a42e8e85776acb1)
• —path - str (file path inside the repository, example: /core/src/components/tab-bar/usage/javascript.md)
• —length - i64 (file length in bytes, example: 529)

3. file_contents

The contents of all non-binary files (i.e. not images/media/data).

• —blob_id - str (Software Heritage file blob id, example: 47953673b7b51c2585402a91d434f5fe4d9dc105)
• —content - str (file contents, always a UTF-8 string)
• —srcencoding - str (original file's encoding, example: `UTF-8`) language - str (programming language label, example: `Python`) isvendor - bool (True if it's a vendor file, e.g. 3rd party library) is_generated - bool (True if the file is auto-generated)