CultriX/Generate-Knowledge-Graphs
3
1�
2 o�oh� � �\ � d dl Z d dlZd dlmZmZmZ d dlZd dlmZ d dl m3Z4 G d� d� Z y)� N)�List�Dict�Any)�Document)�Configc 5 � � e Zd Zd� Zdedefd�Zdedefd�Zdedefd�Zdedefd�Z dedefd�Z6dedefd �Z d7ede e fd �Z
dd e e d
ede eeef fd�Zy)�DocumentProcessorc �" � t � | _ y )N)r �config)�selfs �IC:\Users\jesse\Documents\Cline\knowledge-graphs\src\document_processor.py�__init__zDocumentProcessor.__init__ s � � �h�� � � file_path�returnc �x � t j j |� dz }|| j j k S )z$Validate file size is within limits.i )�os�path�getsizer �MAX_FILE_SIZE_MB)r r � file_size_mbs r
�validate_file_sizez$DocumentProcessor.validate_file_size s. � ��w�w���y�1�[�A� ��t�{�{� ;� ;�;�;r c � � | j |� s#t d| j j � d�� �t j8 j
|� d j � }|dk( r| j |� S |dk( r| j |� S |dk( r| j |� S |dk( r| j |� S t d|� �� �) z.Load document content based on file extension.zFile size exceeds zMB limit� z.pdfz.docxz.txtz.jsonzUnsupported file format: ) r �9ValueErrorr r r r �splitext�lower� _load_pdf�10_load_docx� _load_txt�11_load_json)r r �file_exts r
�
load_documentz DocumentProcessor.load_document s� � ��&�&�y�1� � 1�$�+�+�2N�2N�1O�x� X�Y� Y��7�7�#�#�I�.�q�1�7�7�9�� �v� ��>�>�)�,� ,�
��
��?�?�9�-� -�
��
��>�>�)�,� ,�
��
��?�?�9�-� -� � 8��12� C�D� Dr c � � d}t j |� 5 }|j D ] }|j � }|s�||dz z
}� ddd� |S # 1 sw Y |S xY w)zLoad PDF content.� �13N)�14pdfplumber�open�pages� extract_text)r r �text�pdf�page� page_texts r
r zDocumentProcessor._load_pdf# s\ � ���
�_�_�Y�
'�3�� � �� � -� -� /� � ��I�� ,�,�D� "� (�15 � � (�16 � �s �"A�17A�Ac �f � t |� }d}|j D ] }||j dz z
}� |S )zLoad DOCX content.r% r& )r �18paragraphsr+ )r r �docr+ � paragraphs r
r z DocumentProcessor._load_docx- s7 � ��y�!���� � � �I� �I�N�N�T�)� )�D� (�� r c �j � t |dd�� 5 }|j � cddd� S # 1 sw Y yxY w)zLoad TXT content.�r�utf-8��encodingN)r( �read)r r �files r
r zDocumentProcessor._load_txt5 s% � �
�)�S�7�
3�t��9�9�;� 4�
3�
3�s �)�2c � � t |dd�� 5 }t j |� }t j |d�� cddd� S # 1 sw Y yxY w)z&Load JSON content and convert to text.r4 r5 r6 � )�indentN)r( �json�load�dumps)r r r9 �datas r
r! z DocumentProcessor._load_json: s7 � �
�)�S�7�
3�t��9�9�T�?�D��:�:�d�1�-� 4�
3�
3�s �,A�Ar+ c �X � t |� | j j k r|gS g }d}|t |� k r�|| j j z }|t |� k rl|j d||� }|dk( r|j d||� }|dk( r|j d||� }|dk7 r$||| j j dz z kD r|dz }||| j � }|r|j |� || j j z19 }|t |� k\ r |S |t |� k r��|S )z2Split text into overlapping chunks for processing.r �.������!�?r; r )�lenr �20CHUNK_SIZE�rfind�strip�append�
CHUNK_OVERLAP)r r+ �chunks�start�end� sentence_end�chunks r
�21chunk_textz DocumentProcessor.chunk_text@ s* � � �t�9�� � �.�.� .��6�M������c�$�i� ��$�+�+�0�0�0�C� �S��Y�� #�z�z�#�u�c� :� � �2�%�#'�:�:�c�5�#�#>�L� �2�%�#'�:�:�c�5�#�#>�L� �2�%�,�����AW�AW�[\�A\�9\�*\�&��*�C���s�O�)�)�+�E���
�
�e�$��$�+�+�3�3�3�E���D� �!���
�/ �c�$�i� �. �
r �22file_paths�23batch_modec � � g }|D ]@ } | j |� }| j |� }|j |||dd�� |s |S �B |S # t $ r*}|j |dg dt |� d�� Y d}~�sd}~ww xY w)zProcess multiple documents.�success)r �contentrL �statusr% �error)r rV rL rW rX N)r# rQ rJ � Exception�str)r rR rS �resultsr rV rL �es r
�process_documentsz#DocumentProcessor.process_documentsa s� � ���#�I�
� �,�,�Y�7�� � � ��1�����!*� &� $� '� � � "�� �� "� $�0 ��� �
����!*� !� � %� ��V� � � ��
�s �:A24�25 A=� A8�8A=N)F)�__name__�26__module__� __qualname__r rZ �boolr r# r r r r! r rQ r r r] � r r
r r s� � � �<�C� <�D� <�27E�s� E�s� E�$�3� �3� ��C� �C� � �3� �3� �28.�C� .�C� .� �s� �t�C�y� �B �D��I� �4� �TX�Y]�^a�cf�^f�Yg�Th� r r ) r r= �typingr r r r' �docxr �config.settingsr r rb r r
�<module>rf s& �� � � "� "� � � "�u� ur