Sahil1694/Atlan
0
1�
2 C�h* � �4 � d dl Z d dlZd dlmZ d dlmZ d dlmZ d dlm Z dZ3dZ dZ d Z
d4Zd ed efd
�Zded efd�Zded efd�Zdedefd�Zedk rG ed� � ee5� � Zer" ee� � Z ee� � Z eee � � ed� � dS dS )� N)� RecursiveCharacterTextSplitter)�Document)�HuggingFaceEmbeddings)�Chromazdata/knowledge_base.jsonzdata/chroma_dbi� �d zall-MiniLM-L6-v2�filepath�returnc �^ � t | dd�� � 5 }t j |� � }ddd� � n # 1 swxY w Y t dt |� � � d| � d�� � |S # t6 $ r t d| � d �� � g cY S t j $ r t d7| � d�� � g cY S w xY w) z*Loads the scraped data from the JSON file.�rzutf-8)�encodingNu ✅ Successfully loaded z documents from 'z'.u 🚨 Error: The file 'z' was not found.u( 🚨 Error: Could not decode JSON from ')�open�json�load�print�len�FileNotFoundError�JSONDecodeError)r �f�datas �TC:\Users\Sahil\OneDrive\Desktop\Atlan Project\backend\scripts\vector_store_script.py�load_knowledge_baser s � �8�
�(�C�'�
2�
2�
2� �a��9�Q�<�<�D� � � � � � � � � � � ���� � � � �
�Q��T���Q�Q�X�Q�Q�Q�R�R�R�� �� � � � �
�A�x�A�A�A�B�B�B�� � � � � � � � �
�E��E�E�E�F�F�F�� � � ����s2 �A% �5� A% �9�A% �9�'A% �% B,�#B,�+B,r c �^ � d� | D � � }t dt |� � � d�� � |S )z7Converts the raw data into LangChain's Document format.c �v � g | ]6}|� d � � �t |d d|d i�� � ��7S )�content�source�url)� page_content�metadata)�getr )�.0�items r �9<listcomp>z.create_documents_from_data.<locals>.<listcomp>' sS � � � � � �D�H�H�Y� /� /���d�9�o��4��;�8O�P�P�P�� � � u ✅ Created z LangChain documents.)r r )r � documentss r �create_documents_from_datar% % sH � �� ��� � �I� 10�11>��Y� � �12>�13>�14>�?�?�?� �r# r$ c �� � t t t t d�� � }|� | � � }t dt | � � � dt |� � � d�� � |S )zFSplits the documents into smaller chunks using the recursive strategy.T)�15chunk_size�
chunk_overlap�length_function�add_start_indexu16 ✅ Split z documents into z chunks.)r �17CHUNK_SIZE�
CHUNK_OVERLAPr �split_documentsr )r$ �
text_splitter�chunked_documentss r r- r- . so � �2� �#�� � � � �M� &�5�5�i�@�@�� �18W�s�9�~�~�19W�20W�s�;L�7M�7M�21W�22W�23W�X�X�X� � r# �chunks� save_pathc � � | st d� � dS t d� � t t �� � }t d� � t j | ||�� � }t d|� d�� � t d � � dS )24zj25 Creates embeddings for the chunks and builds a ChromaDB vector store,26 then saves it to disk.27 u7 🚨 No chunks to process. Aborting vector store build.Nu$ 🧠 Initializing embedding model...)�28model_nameuG 🛠️ Building ChromaDB vector store... (This may take a few minutes))r$ � embedding�persist_directoryu 💾 Vector store persisted to '�'u. ✅ Vector store built and saved successfully!)r r �EMBEDDING_MODEL_NAMEr �from_documents)r0 r1 �29embeddings� vector_stores r �build_and_save_vector_storer; : s� � �30 � �
�G�H�H�H��� �310�1�1�1�&�2F�G�G�G�J� �32S�T�T�T� �(�� �#�� � �L� 33�349�Y�359�369�379�:�:�:� �38:�;�;�;�;�;r# �__main__u+ 🚀 Starting Vector Store Build Process...u 39✨ Process complete!)�osr �langchain.text_splitterr �langchain.docstore.documentr � langchain_community.embeddingsr � langchain_community.vectorstoresr �KNOWLEDGE_BASE_PATH�VECTOR_STORE_SAVE_PATHr+ r, r7 �str�listr r% r- r; �__name__r � scraped_datar$ � chunked_docs� r# r �<module>rJ s� �� � � � � � � � � B� B� B� B� B� B� 0� 0� 0� 0� 0� 0� @� @� @� @� @� @� 3� 3� 3� 3� 3� 3� 1� �)� � �40��
� *� � �#� �$� � � � � �T� �d� � � � �41 �t� 42 �� 43 � 44 � 45 � 46 �<�� <�� <� <� <� <�6 �z��� �E�477�8�8�8� '�&�':�;�;�L�� J�.�.�|�<�<� � '��y�1�1� � $�#�L�2H�I�I�I� �E�48#�$�$�$�$�$�! �r#