LyraNovaHeart/TheRoyalCarpet-Deduped-Compressed
The Royal Carpet: Cleaned Original Dataset: KaraKaraWitch/the-royal-carpet Thanks KaraKaraWitch! Why? The original dataset is 40GB in size (at least in decimal), which is quite the large set! 40GB would likely overwhelm most systems since I doubt most people have more than 32GB of RAM. Not to mention too, this dataset contained a redundant row, html, which is just a exact copy of the text row with HTML crap, removing that row brought down most of the dataset to… See the full description on the dataset page: https://huggingface.co/datasets/LyraNovaHeart/TheRoyalCarpet-Deduped-Compressed.
The Royal Carpet: Cleaned
Original Dataset: KaraKaraWitch/the-royal-carpet
Thanks KaraKaraWitch!
Why?
The original dataset is 40GB in size (at least in decimal), which is quite the large set! 40GB would likely overwhelm most systems since I doubt most people have more than 32GB of RAM. Not to mention too, this dataset contained a redundant row, html, which is just a exact copy of the text row with HTML crap, removing that row brought down most of the dataset to roughly... 15.2GB. I did check the dataset too, it looks like it cleanly removed just the HTML row, which means nothing else besides that got modified, as a side thing, I also compressed the set into 7-Zip archives to speed up the download process, which reduces it to just 4GB overall.
Download Links (For Convience)
JSONL Part 1: Download Part 2: Download Part 3: Download Part 4: Download
Total Size: 15.06GB (Binary), 16.17GB (Decimal)
7z
Part 1: Download Part 2: Download Part 3: Download Part 4: Download
Total Size: 3.81GB (Binary), 4.00GB (Decimal)
Code Used to Remove the Row
import json
def remove_columns_from_jsonl(input_file, output_file, columns_to_remove):
"""
Remove specified columns from JSONL file.
Args:
input_file: Input JSONL file path
output_file: Output JSONL file path
columns_to_remove: List of column names to remove (e.g., ['html', 'other_column'])
"""
with open(input_file, 'r', encoding='utf-8') as infile, \
open(output_file, 'w', encoding='utf-8') as outfile:
for line_num, line in enumerate(infile, 1):
try:
data = json.loads(line.strip())
# Remove specified columns
for column in columns_to_remove:
if column in data:
del data[column]
# Write cleaned data
outfile.write(json.dumps(data, ensure_ascii=False) + '\n')
except json.JSONDecodeError as e:
print(f"Error on line {line_num}: {e}")
print(f"✓ Processed {line_num} lines")
print(f"✓ Removed columns: {columns_to_remove}")
# Usage:
remove_columns_from_jsonl('TheRoyalCarpet-split-00.jsonl', 'cleaned-001.jsonl', ['html'])```
# Credit
KaraKaraWitch: [Profile](https://huggingface.co/KaraKaraWitch)