Apppublic

liuyimeta/training_data_chat

sourceHugging Faceapache-2.0updated 3y agoView on Hugging Face

0likes

training.cpython-39.pyc13 linesDownload Raw Back to __pycache__

1a

2.wd��@sTddlZddlmZddlmZddlmZddlmZddl	Z	ddl3Z4dd�ZdS)�N)�FAISS)�Path)�CharacterTextSplitter)�OpenAIEmbeddingsc		Cstt|��d��}t|�dkr0tdtjd�dSg}|D]R}t|ddd��0}td|j�d	��|�	|�5��Wd�q81s�0Yq8td6ddd
�}g}|D]}|�|�
|��q�t�|t��}t�|jd�d|_tdd��}t�||�Wd�dS1�s70YdS)Nz**/*.*�zQThe folder training/facts should be populated with at least one .txt or .md file.)�file�rzutf-8)�encodingzAdd z to dataseti��8r)�9chunk_size�	separator�
chunk_overlapzafter_training/training.indexzafter_training/faiss.pkl�wbu训练完成)�listr�glob�len�print�sys�stderr�open�name�append�readr�extend�10split_textr�11from_textsr�faiss�write_index�index�pickle�dump)	�12files_path�trainingData�data�training�f�textSplitter�docs�setsZstore1�r)�4/Users/kai/PycharmProjects/TTS/liuyimeta/training.py�train13s&.r+)rZlangchain.vectorstoresr�pathlibr�langchain.text_splitterrZlangchain.embeddingsrrrr+r)r)r)r*�<module>s