unum-cloud/ann-codesearch-4m

Cleaning Unlike the original dataset, the func_code_string column was updated to remove any comments and keep just the code. The original version can still be found in the whole_func_string. import re def remove_comments_docstrings(code, language): if language == 'python': # Remove docstrings code = re.sub(r'"""(.*?)"""', '', code, flags=re.DOTALL) code = re.sub(r"'''(.*?)'''", '', code, flags=re.DOTALL) # Remove comments code =… See the full description on the dataset page: https://huggingface.co/datasets/unum-cloud/ann-codesearch-4m.