neil-code/dialogsum-test
Dataset Card for DIALOGSum Corpus Dataset Description Links Homepage: https://aclanthology.org/2021.findings-acl.449 Repository: https://github.com/cylnlp/dialogsum Paper: https://aclanthology.org/2021.findings-acl.449 Point of Contact: https://huggingface.co/knkarthick Dataset Summary DialogSum is a large-scale dialogue summarization dataset, consisting of 13,460 (Plus 100 holdout data for topic generation) dialogues with… See the full description on the dataset page: https://huggingface.co/datasets/neil-code/dialogsum-test.
15234
1import csv2import json3 4 5def crop_json(file_path, num_lines, new_file_path):6 with open(file_path, 'r') as file:7 data = json.load(file)8 9 cropped_data = data[:num_lines]10 11 with open(new_file_path, 'w') as new_file:12 json.dump(cropped_data, new_file)13 14 15# file_paths = [16# {'input': './data/corpus/train.json', 'size': 2000, 'output': './data/corpus-cut/train.json'},17# {'input': './data/corpus/test.json', 'size': 500, 'output': './data/corpus-cut/test.json'},18# {'input': './data/corpus/val.json', 'size': 500, 'output': './data/corpus-cut/val.json'}19# ]20 21#22# for file_path in file_paths:23# crop_json(file_path['input'], file_path['size'], file_path['output'])24 25 26def crop_csv(file_path, num_lines, new_file_path):27 with open(file_path, 'r') as file:28 reader = csv.reader(file)29 data = [row for row in reader]30 31 cropped_data = data[:num_lines]32 33 with open(new_file_path, 'w') as new_file:34 writer = csv.writer(new_file)35 writer.writerows(cropped_data)36 37 38file_paths = [39 {'input': './train.csv', 'size': 2000, 'output': './train.csv'},40 {'input': './test.csv', 'size': 500, 'output': './test.csv'},41 {'input': './validation.csv', 'size': 500, 'output': './validation.csv'}42]43 44for file_path in file_paths:45 crop_csv(file_path['input'], file_path['size'], file_path['output'])46