ekatra/Mobius

1o

2� �c��@spdd
lm
Z

ddlZddlmZ
ddlZddlmZ
dd�Z	dd�Zd	d3�Z4dd�Zd
d�Zdd�Z
dd�ZdS)�)
�normN)
�SentenceTransformer)
�
sent_tokenizec
Cs>t�
|�
}
g}tt|
j�
�
D]}|
j|}|�|���

q|S)
z�5    This function reads the pdf file and extracts the text from it.6 7    Parameters:8    fname (str): Name of the pdf file9 10    Returns:11    text_ext (list): List of extracted text from the pdf file12    )�PyPDF2Z	PdfReader�range�len�pages�appendZextract_text)�fname�reader�text_ext�
iZpageObj�r�DD:\Microsoft Codes\NLP\Chatbots\HuggingFace\Ekatra-QnA\operations.py�read_pdfs13 14

15
rc
Cs.g}
|D]	}|
�t
|�
�

qtd
t|
�
�
|
S)z�16    This function apply sent_tokenize to the text and stores the result in a list.17 18    Parameters:19    text_ext (list): List of extracted text from the pdf file20 21    Returns:22    sent_toks (list): List of tokenized sentences23    zlen(sent_toks) )r	r�printr)r�	sent_toksr
rrrrs2425

rc
C�td
�
}
|
�
|�
}|S)z�26    This function creates embeddings for the document sentences.27 28    Parameters:29    concat_list (list): List of tokenized sentences30 31    Returns:32    embeddings (list): List of embeddings of the sentences33    �&sentence-transformers/all-MiniLM-L6-v2�r�encode)�concat_list�model�34embeddingsrrr�create_content_embeddings/s35 36
rc
Cr)z�37    This function creates embeddings for the query.38    Parameters:39    query_text (str): Query entered by the user40 41    Returns:42    query_embedding (list): List of embeddings of the query43    rr)Z44query_textr�query_embeddingrrr�create_query_embeddings?s45 46

rcCsHg}tt
|�
�
D]}t�||
|�t|�
t|
|�
}|�|�

q|S)
a\
47 48    This function calculates cosine similarity between the query and the sentences.49 50    Parameters:51    query_embedding (list): List of embeddings of the query52    embeddings (list): List of embeddings of the sentences53    concat_list (list): List of tokenized sentences54 55    Returns:56    cosine_lis (list): List of cosine similarity values57    )rr�np�dotrr	)rrr�58cosine_lisr
Zcosinerrr�calculate_cosineNs


�
�r cs�|�tt
t��
�
�f
d
d�dd�d|
�}td|�
t
t��
�
}t|�f
dd�dd�}g}t
|
�
D]	}|�||�

q1t|�

|S)z�59    This function fetches the top N ranked sentences.60 61    Parameters:62    cosine_lis (list): List of cosine similarity values63    N (int): Number of sentences to be ranked64 65    Returns:66    indexes_final (list): List of top N ranked sentences67    c

��|S�
Nr�
r
�
�list1rr�<lambda>t�z$fetch_top_rank_ans.<locals>.<lambda>T��key�reverseNzindexes_final c

r!r"rr#r$rrr&yr')�sortedrrrr	)r�
N�
indexes_final�indices�sorted_indicesr
rr$r�fetch_top_rank_ansfs

�
�68




r0cs�i}|}|D]}|
|||<q|g
}t|�

g}d
}	g}69t
t��
�
}t|�f
dd�dd�}t
t|�
�
D]}
|	dkr;
n|�|
||
�dd��

|70�t|
�
�

q3td	t|�
�d71��

d}|d�|�
d
|d}|S)a�
72    This function fetches the most relevant sentences, pass it as a context to GPT-3 prompt along with user's query.73 74    Parameters:75    indexes_final (list): List of top N ranked sentences76    concat_list (list): List of tokenized sentences77    list1 (list): List of cosine similarity values78    query (str): Query entered by the user79 80    Returns:81    prompt (str): GPT-3 prompt82    r
c

r!r"rr#r$rrr&�r'z%fetch_most_relevant.<locals>.<lambda>Tr(i�
�
83�
 z	Selected z document sections:z�Answer the question as truthfully as possible using the provided context, and if the answer is not contained within the text below, say "I don't know."84 85Context:86�z87 88 Q: z89 A:)rrr+r	�replace�strr�join)r-rr%�query�dicts�keysr
Zmost_relevant_document_sectionsZchosen_sectionsZchosen_sections_lenZchosen_sections_indexesr.r/Z
section_index�header�promptrr$r�fetch_most_relevant�s,







�
r<)�numpy.linalgr�numpyrZsentence_transformersrr�
nltk.tokenizerrrrr r0r<rrrr�<module>
s