dnagpt/human_genome_GCF_009914755.1

Dataset Card for "humangenomeGCF_009914755.1"

how to build this data: human full genome data from: https://www.ncbi.nlm.nih.gov/datasets/genome/GCF_009914755.1/

Preprocess: 1 download data use ncbi data set tools: curl -o datasets 'https://ftp.ncbi.nlm.nih.gov/pub/datasets/command-line/LATEST/linux-amd64/datasets' chmod +x datasets ./datasets download genome accession GCF000001405.40 --filename genomes/humangenome_dataset.zip then move the gene data to human2.fra

2 write the origin data into pure dna data, one line 1000 bp/letters:

filename = "human2.fna"
data_file = open(filename, 'r')

out_filename = "human2.fna.line"
out_file = open(out_filename, 'w')

max_line_len = 1000 #1000个字母一行数据

text = ""
for line in data_file:
    line = line.strip()
    if line.find(">") != 0: #去掉标题行
        line = line.upper()
        line = line.replace(" ","").replace("N","") #去掉N和空格
        text = text + line
        if len(text) > max_line_len:
            text = text.strip()
            out_file.write(text+"\n")
            text = "" #clear text
#last line
if len(text) <= max_line_len:
    pass #不要了


3 split data into train and valid dataset:

filename = "human2.fna.line" data_file = open(filename, 'r')

outtrainfilename = "human2.fna.line.train" outtrainfile = open(outtrainfilename, 'w')

outvalidfilename = "human2.fna.line.valid" outvalidfile = open(outvalidfilename, 'w')

linenum = 0 selectline_num = 0

for line in datafile: if 0==linenum%3: #取1/3数据 if selectlinenum%100: #取1%做校验 outtrainfile.write(line) else: outvalidfile.write(line) selectlinenum = selectlinenum + 1

linenum = linenum + 1

4 the we could use it in local, or push it to hub:

from datasets import load_dataset
data_files = {"train": "human2.fna.line.train", "test": "human2.fna.line.valid"}
train_dataset = load_dataset("text", data_files=data_files)
train_dataset.push_to_hub("dnagpt/human_genome_GCF_009914755.1",token="hf_*******")




[More Information needed](https://github.com/huggingface/datasets/blob/main/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)