CoolFace
Apppublic

norasalem88/rag_assistant_devops

sourceHugging Faceupdated 7mo agoView on Hugging Face
0likes
ingestion.cpython-313.pyc67 linesDownload Raw Back to __pycache__
12(�i�!���SSKrSSKrSSKJr SSKJr SSKJr SSKJ	r	 SSK3Jr SSKJ
r
 \
"\5r\R R#\R R#\R R%\555r\R R+\5S:wa+\R R-\R."5S5r\R R-\S	5r\R R-\S45rSrS\S
\4SjrS\S
\4SjrS\S
\4SjrS
\ 4Sjr!S\4Sjr"S\S
\4Sjr#Sr$\S:Xa\$"5 gg)�N)�PyPDFLoader)�RecursiveCharacterTextSplitter��HuggingFaceEmbeddings)�Chroma)�Document)�5get_loggerzdevops-rag-assistant�data�vectorstore_v3c��[SS9$)Nzall-MiniLM-L6-v2)�6model_namer���FE:\Devops chatbot\Devops-chatbot\devops-rag-assistant\src\ingestion.py�get_embeddingsrs�� �,>�?�?r�pdf_path�returnc	��SSKJn SSKn[RRSS5n[RRUSSS5n[RRUSS	S75n[RRU5(aXBRl	[RRU5(aU[RS'[RSU35 U"US
S9n/n[U5H�up�URU85nUR!5(aE[#UUU	S.S9nUR%U5 [RSU	S['U5S35 Mp[R)SU	S35 M� [RS[RR+U5S['U5S35 U$![a'n[RSUSU35 /sSnA$SnAff=f)z�9OCR fallback for image-based PDFs.10Converts each page to an image using pdf2image, then runs Tesseract OCR.11Returns a list of Document objects with page_content and metadata.12r)�convert_from_pathN�CONDA_PREFIX��Library�binz
tesseract.exe�share�tessdata�TESSDATA_PREFIXzRunning OCR on: i,)�dpizpdf2image failed for �: )�source�page)�page_content�metadataz  OCR page z chars extractedz: no text foundzOCR complete for z pages with text)�	pdf2imager�pytesseract�os�environ�get�path�join�exists�
tesseract_cmd�logger�info�	Exception�error�	enumerate�image_to_string�stripr�append�len�warning�basename)
rrr$�conda_prefix�
tesseract_bin�tessdata_dir�images�e�ocr_docs�page_num�img�text�docs
             r�ocr_pdfrAs���,���:�:�>�>�.�"�5�L��G�G�L�L��y�%��Q�M��7�7�<�<��i��*�M�L�	�w�w�~�~�m�$�$�0=���-�	�w�w�~�~�l�#�#�(4��1314�$�%�15�K�K�"�8�*�-�.��"�8��5��16�H�"�6�*�
���*�*�3�/���:�:�<�<��!�&�$���C�
�O�O�C� ��K�K�+�h�Z�r�#�d�)��<L�M�N��N�N�[��17�/�B�C�+��K�K�#�B�G�G�$4�$4�X�$>�#?�r�#�h�-��P`�a�b��O��+�����,�X�J�b���<�=��	���s�3G�18H	�"H�>H	�H	�data_dirc���/n[R"U5Vs/sH*o"R5RS5(dM(UPM, nnU(d[RSUS35 /$UH�n[RRX5n[RSU35 [USS9nUR5n[S19U55n	[RS[U5SU	S
35 U	S:Xa$[RSUS35 [U5nUR!U5 M� U$s snf![a'n[RSUSU35 /nS	nAN�S	nAff=f)z�20Load all PDFs from data_dir. For each PDF:211. Try PyPDFLoader (text extraction).222. If total extracted chars == 0, fall back to OCR.23�.pdf�No PDFs found in �.�Processing: T��extract_images�PyPDFLoader failed for rNc3�h# �UH(n[URR55v� M* g7f�N�r4r!r2��.0�ds  r�	<genexpr>� load_all_pdfs.<locals>.<genexpr>`s%���D�t�!�#�a�n�n�2�2�4�5�5�t���02�  PyPDFLoader: � pages, � charsr�  �< appears to be image-based (0 chars). Falling back to OCR...)r%�listdir�lower�endswithr,r5r(r)r-r�loadr.r/�sumr4rA�extend)24rB�all_docs�f�	pdf_files�filenamer�loader�docsr;�total_charss25          r�
load_all_pdfsrfGsG���H��J�J�x�0�O�0�q�G�G�I�4F�4F�v�4N��0�I�O�����*�8�*�A�6�7��	����7�7�<�<��3�����l�8�*�-�.�	� ��$�?�F��;�;�=�D�26�D�t�D�D�����o�c�$�i�[���
�V�L�M��!���N�N�R��z�)e�f�g��8�$�D������)�,�O��9P���	��L�L�2�8�*�B�q�c�B�C��D��	�s#�'D?�D?�-E�27E5�E0�0E5�filepathc���[R"5n[US5nURUR	55 SSS5 UR5$!,(df   UR5$=f)zCompute MD5 hash of a file.�rbN)�hashlib�md5�open�update�read�	hexdigest)rg�hasherr`s   r�_compute_file_hashrqmsW��
�[�[�]�F�	
�h��	���
�
�a�f�f�h��28������29�	������s� A�30A7c���URRS/S9nUSn0nUH&nU(dMSU;dMSU;dMUSX4S'M( U$![a 0s$f=f)zU31Returns a dict of {doc_title: doc_id} for all documents already in the vectorstore.32�	metadatas)�include�	doc_title�doc_id)�_collectionr'r.)�vectorstore�itemsrs�existing�ms     r�_get_existing_doc_infor|us|����'�'�+�+�[�M�+�B���+�&�	��H�
���1���!�h�!�m�'(��{�H�{�^�$���O�����	��s�A�A �A ruc�8�URRSU0/S9nUSnU(a?URRUS9 [R	S[U5SUS35 g33g34![a%n[RSUS	U35 S35nAg36S37nAff=f)z=Remove all chunks for a given doc_title from the vectorstore.ru)�wherert�ids)rz38  Removed z old chunks for '�'z#  Failed to remove old chunks for 'z': N)rwr'�deleter,r-r4r.r/)rxrury�
ids_to_deleter;s     r�_remove_doc_chunksr��s���39N��'�'�+�+��	�*��,�40���e��
���#�#�*�*�}�*�=��K�K�*�S��%7�$8�8I�)��TU�V�W����N����:�9�+�S���L�M�M��N�s�A&A*�*41B�4B�Bc���[RRU5n[R	SU35 [USS9nUR
5n[SU55n[R	S[U5S	US4235 US:Xa$[RSUS
35 [U5nU$![a'n[RSUSU35 /nSnAN�SnAff=f)zK43Load a single PDF. Try text extraction first, fall back to OCR if needed.44rGTrHrJrNc3�h# �UH(n[URR55v� M* g7frLrMrNs  rrQ�"load_single_pdf.<locals>.<genexpr>�s%���@�4�a�c�!�.�.�.�.�0�1�1�4�rSrTrUrVrrWrX)
r%r(r6r,r-rr\r.r/r]r4r5rA)rrbrcrdr;res      r�load_single_pdfr��s����w�w����)�H�45�K�K�,�x�j�)�*���X�d�;���{�{�}��46�@�4�@�@�K�47�K�K�/�#�d�)��H�[�M��H�I��a������H�:�%a�b�c��x� ���K�������.�x�j��1�#�>�?�����s�B6�648C'�C"�"C'c��[RS[35 [RR[5(d[R"[5 [5n[[US9n[U5n[RS[UR5535 [R"[5Vs/sH*o3R5RS5(dM(UPM, nnU(d[R!S[S35 g/nUGHon[RR#[U5n[%U5nXb;aHX&U:XaSUS	35 M[[RS49US35 ['X5 O[RSUS
35 [)U5n	U	(d!SUS35 M�[+SSSS9n50U51R-U	5nUVs/sH%o�R.R15(dM#UPM' nnUHnX�R2S'XlR2S'M! [RS[5U5SUS35 UR7U5 GMr U(d[RS5 g[RS[5U5S35 UR9U5 [RS5 gs snfs snf)z�52Incremental ingestion:53- Opens existing vectorstore (or creates a new one).54- Checks which doc_titles are already indexed and their doc_id (hash).55- Only processes NEW or CHANGED PDFs.56- Removes old chunks for changed PDFs before adding new ones.57zLoading PDFs from )�persist_directory�embedding_functionz#Existing documents in vectorstore: rDrErFNzSKIP 'u%' — already indexed with same hash.zUPDATE 'u2' — file changed (hash mismatch). Re-indexing...zNEW 'u ' — not yet indexed. Adding...z  No text extracted from 'z'. Skipping.i���T)�58chunk_size�
chunk_overlap�add_start_indexrvruz59  Created z
 chunks for 'r�z5No new documents to index. Vectorstore is up to date.zAdding z! new chunks to the vectorstore...z)Ingestion complete. Vector store updated.)r,r-�DATA_DIRr%r(r*�makedirsrr�VECTORSTORE_DIRr|�list�keysrYrZr[r5r)rqr�r�r�split_documentsr!r2r"r4r^�
add_documents)
�60embeddingsrx�
existing_docsr`ra�61new_splitsrbr�current_hashrd�
text_splitter�splits�ss
             r�ingest_documentsr��sv���K�K�$�X�J�/�0�
�7�7�>�>�(�#�#�62���H���!�J��)�%��K�+�;�7�M�63�K�K�5�d�=�;M�;M�;O�6P�5Q�R�S��J�J�x�0�O�0�q�G�G�I�4F�4F�v�4N��0�I�O�����*�8�*�A�6�7���J����7�7�<�<��(�3��)�(�3���$��&�,�6����f�X�J�.S�T�U�����h�x�j�0b�c�d�"�;�9��K�K�%��z�)I�J�K��x�(����N�N�7��z��N�O��7��� �64�
�65�.�.�t�4��#�>�V��~�~�';�';�'=�!�V��>��A�#/�J�J�x� �&.�J�J�{�#��	���j��V��
�]�8�*�A�F�G����&�!�G�J����K�L��66�K�K�'�#�j�/�*�*K�L�M����j�)�67�K�K�;�<��cP��B?s�
'L�8L�"L�5L�__main__)%r%rj�$langchain_community.document_loadersr�langchain_text_splittersr�langchain_huggingfacer� langchain_community.vectorstoresr�langchain_core.documentsr�	src.utilsr	�__name__r,r(�dirname�abspath�__file__�BASE_DIRr6r)�getcwdr�r�r�strr�rArfrq�dictr|r�r�r�rrr�<module>r�s;��	��<�C�7�3�-� �	�H�	���7�7�?�?�2�7�7�?�?�2�7�7�?�?�8�+D�E�F���7�7���H��!7�7��w�w�|�|�B�I�I�K�)?�@�H�
�7�7�<�<��&�)���'�'�,�,�x�)9�:��@�+�c�+�d�+�\#�C�#�D�#�L������4��"N�s�N��c��d��0J=�X�z����r