applied-ai-018/peacock-data-public-datasets-idc-enfm-dataprocessing

BharatGPT Data Curation for English Foundational Model RedPajamaV2 RedPajamaV2 directory contains scripts for processing and filtering of RedPajamaV2 dataset. Overall pipeline is as follows: URL Filtering scripts/utils/url_filtering.py: The first step of filtering is URL based. We use a blocklist of URLs known to contain inappropriate content. This list is taken from blocklistproject. It contains categories for advertisements, gambling, adult… See the full description on the dataset page: https://huggingface.co/datasets/applied-ai-018/peacock-data-public-datasets-idc-enfm-dataprocessing.