Dwild/ChuanhuChatGPT
0
1a
2 � ,dj � @ s� d dl Z d dlZd dlmZ d dlmZmZmZmZmZ d dl Z d dl3Z4d dl m Z d dl T d dl
T dd� Zdd � Zd5d � Zddd�Zdd� ZdS )� N)�download_loader)�Document� LLMPredictor� PromptHelper�QuestionAnswerPrompt� RefinePrompt)�tqdm)�*c C s| dd� | D �}|j dd� d� t�� }|D ]H}t|d��*}|�d� }rT|�|� q:W d � q*1 sh0 Y q*|�� S )Nc S s g | ]6}|j �qS � )�name)�.0�xr7 r8 �K/Users/treediagram/Documents/Developer/ChuanhuChatGPT/modules/llama_func.py�9<listcomp> � z"get_index_name.<locals>.<listcomp>c S s t j�| �S )N)�os�path�basename)r
r10 r11 r �<lambda> r z get_index_name.<locals>.<lambda>)�key�rbi )�sort�hashlib�md5�open�read�update� hexdigest)�file_src�12file_pathsZmd5_hash� file_path�f�chunkr13 r14 r �get_index_name s ,r# c C s8 g }t | �dkr4|�t| d d� �� | dd � } q|S )Nr i� )�len�appendr )�text�blocksr15 r16 r � block_split s17 r( c C s g }t �d� t �d| � �� | D �]�}|j}tj�|�}tj�|�d }t �d|� �� |dk�rt �d� z8ddlm } dd l18m } |d19 � d d �}|||�j
} W n` d
} t|d��6}20t�|21�} t| j�D ]} | | �� 7 } q�W d � n1 s�0 Y Y n0 | }
n�|dk�rDt �d� td�}|� }|j|d�d j
}
n�|dk�rzt �d� td�}|� }|j|d�d j
}
n`|dk�r�t �d� t|�}
nBt �d� t|ddd��}|�� }
W d � n1 �s�0 Y t|
�}|t|�g7 }q"t �d � |S ) NzLoading documents...z22file_src: � zloading file: z.pdfzLoading PDF...r )� parse_pdf)� advance_docs�pdf�23two_columnF� r z.docxzLoading Word...�24DocxReader)�filez.epubzLoading EPUB...�25EpubReaderz.xlsxzLoading Excel...zLoading text file...�rzutf-8)�encodingzDocuments loaded.)�logging�debugr r r r �splitext�infoZmodules.pdf_funcr* �modules.configr+ �getr& r �PyPDF2� PdfReaderr �pages� extract_textr � load_data�excel_to_stringr � add_spacer )r � documentsr0 �filepath�filename� file_typer* r+ r- ZpdftextZ26pdfFileObjZ pdfReader�page�text_rawr/ �loaderr1 r! r& r27 r28 r �
get_documents& sT 2930 3132 3323435363738394041(42rH � � � �X � c 43 C s� ddl m} ddlm} m}44 | tjd< |dkr2d n|}|dkrBd n|}|dkrRdn|}t|d| d�d �} t||||d45|d �} t |�}
tj46� d |
� d
��r�t �
d� | �d |
� d
��S z�t|�}t �
d� t� �. |47j| | |d�}| j||d�}W d � n1 �s0 Y t �d� tjddd� |�d |
� d
�� t �d� |W S t�y� } z"t �d|� t|� W Y d }~d S d }~0 0 d S )Nr )�48ChatOpenAI)�GPTSimpleVectorIndex�ServiceContext�OPENAI_API_KEYr. rM zgpt-3.5-turbo-0301)�49model_name�openai_api_key)�llmrL )�max_input_size�50num_output�max_chunk_overlap�embedding_limit�chunk_size_limit� separatorz./index/z.jsonu0 找到了缓存的索引文件，加载中……u 构建索引中……)�
llm_predictor�
prompt_helperrY )�service_contextu 索引构建完成！z./indexT)�exist_oku 索引已保存至本地!u 索引构建失败！)Zlangchain.chat_modelsrN � llama_indexrO rP r �environr r r# r �existsr4 r7 �load_from_diskrH �retrieve_proxy�
from_defaults�from_documentsr5 �makedirs� save_to_disk� Exception�error�print)�api_keyr rU Z num_outputsrW rY rX rZ rN rO rP r[ r\ �51index_namerA r] �index�er52 r53 r �construct_indexV s> 54 5556�5758�&5960 ro c C s4 ddddddd�}|� � D ]\}}| �||�} q| S )Nu ， u 。 u ？ u ！ u ： u ； )u ，u 。u ？u ！u ：u ；)�items�replace)r& Z punctuationsZcn_puncZen_puncr61 r62 r r@ � s r@ )rI rJ rK rL NrM )r r4 r_ r r r r r r �coloramar: r �modules.presets�
modules.utilsr# r( rH ro r@ r63 r64 r65 r �<module> s& 3 �66/