Datasetpublic

bigcode/github-commits-diff-dedup-pjjs-april

Deduplicated Commits Deduplicated based on diff: content = '\n'.join(difflib.unified_diff( old_content.splitlines(keepends=True), new_content.splitlines(keepends=True), n=5 )) Parameters: Minimum ngram size: 5 MinHash ngram size: 5 MinHash threshold: 0.8

sourceHugging Faceupdated 3y agoView on Hugging Face

4likes1.9kdownloads

Dataset Card

Deduplicated Commits

Deduplicated based on diff:

content = '\n'.join(difflib.unified_diff(
    old_content.splitlines(keepends=True), 
    new_content.splitlines(keepends=True), 
    n=5
))

Parameters:

Minimum ngram size: 5 MinHash ngram size: 5 MinHash threshold: 0.8