Sameer360/Document_based_chatbot
0
1a
2 X��g� � @ sb d dl mZmZ d dlmZ d dlmZ d dlmZ d dl Z d dl3m Z m Z edd�Z
d d4� ZdS ) � )� PyPDFLoader�5TextLoader)� RecursiveCharacterTextSplitter)�Chroma)�HuggingFaceEmbeddingsN)�PERSIST_DIRECTORY�
DOCUMENTS_DIRz&sentence-transformers/all-MiniLM-L6-v2)�6model_namec C s. t j�t�r:tttd�}dd� |jdgd��dg �D �}n7d }t� }g }t| |�D ]�\}}||v rrt d|� �� qRt d|� �� t j�8|�d }|d kr�t |�� � }n*|d9kr�t
|d d �� � }nt d
|� �� qRtddd��|�} | D ]}10||11jd< q�|�| � qR|�r*|�r|�|� ntj|ttd�}|S )N)�persist_directory�embedding_functionc S s h | ]}d |v r|d �qS )�source� )�.0�metar
r
�)D:\LangChain\backup\document_processor.py� <setcomp> s �z$process_documents.<locals>.<setcomp>� metadatas)�includez!Skipping already processed file: z Processing � z.pdfz.txtzutf-8)�encodingz Skipping unsupported file type: i� �� )�12chunk_size�
chunk_overlapr )r13 )�os�path�existsr r �14embeddings�get�set�zip�print�splitextr �loadr r �split_documents�metadata�extend�
add_documents�from_documents) �files�15file_names�db�existing_source�16all_chunks� file_path� file_name�ext�doc�chunks�chunkr
r
r �process_documents
sJ �� �� �r3 )�$langchain_community.document_loadersr r �langchain.text_splitterr �langchain_chromar �langchain_huggingfacer r �configr r r r3 r
r
r
r �<module> s 17