Alex22356/PsyQA
The data is originally source from (Sun et al,2021). (Liu et al, 2023) processed the data to make it a dataset vis huggingface api with taining/validation/testing splitting Please cite: @misc{liu2023enhancing, title={Enhancing Long-form Text Generation in Mental Health with Task-adaptive Tokenization}, author={Siyang Liu and Naihao Deng and Sahand Sabour and Yilin Jia and Minlie Huang and Rada Mihalcea}, year={2023}, eprint={2310.05317}, archivePrefix={arXiv}… See the full description on the dataset page: https://huggingface.co/datasets/Alex22356/PsyQA.
031
1import json2import random 3random.seed(42)4def split_data():5 f = open("./PsyQA_full.json")6 lines = json.load(f)7 random.shuffle(lines)8 train_data = lines[:int(len(lines)*0.8)]9 valid_data = lines[int(len(lines)*0.8):int(len(lines)*0.9)]10 test_data = lines[int(len(lines)*0.9):]11 print(len(train_data), len(valid_data), len(test_data))12 with open("train.json", "w") as f:13 json.dump(train_data, f)14 with open("valid.json", "w") as f:15 json.dump(valid_data, f)16 with open("test.json", "w") as f:17 json.dump(test_data, f) 18split_data()