CalebKoster/Translation_Note_Alignment
0
1a
2 yE�e6' � @ s� d dl Z d dlZd dlZd dlmZ d dlmZ d dlZd dlmZm Z m3Z4 d dl m Z d dl
mZ d dlmZ d dlZd dlmZ G d d5� d6�ZdS ) � N)�detect)�TfidfVectorizer)�models�select� instruction)�OpenAI)�uroman)�ScriptureReference)�greek_to_hindic @ s^ e Zd ZejZdZdd� Zdd� Zdd� Z dd � Z7d8d � Z d d
� Z dd� Z
dd� Zdd� ZdS )�TranslationNoteFinderzbibles/grc-grctcgnt.txtc C s� || _ tj|ddd�| _| j| _| jdt� �7 _tj| _| �| j �| _9| �|�| _ | j � � d }t
d|� �� t|�| _t�| j� tj| jdd�| _|d | _| �|�| _| �|�| _ | �� \| _| _t
d10| jd d �d d �f �� � �� d S ) Ni � )Zn_ctxZ n_gpu_layerszBHere are some good examples of translating from Greek into Hindi: �Z zFirst line of NT: �tokenize)ZlangZ11processorsz12.tfidf.pklzportion of tfidf matrix �13 )�translation_notes_pathr ZLlamaCppZllm�lmr14 r �verses�15load_bible�greek_bible_path�greek_bible_text�target_bible_text�16splitlines�printr �language�stanzaZdownloadZPipeline�nlpZtfidf_vectorizer_path�load_translation_notes�translation_notes� create_tfidf_vectorizer_matrix�tfidf_vectorizer� tfidf_matrix�todense)�selfr Zbible_text_pathZ17model_pathZ
first_line_nt� r# �Jc:\Users\caleb\Bible Translation Project\guidance\TranslationNoteFinder.py�__init__ s" 18 19 z TranslationNoteFinder.__init__c C sN t |ddd��,}t�|�}td|d � �� W d � n1 s@0 Y |S )N�r�utf-8��encodingzFirst translation note: r )�open�json�loadr )r" r �filer r# r# r$ r 1 s 200z,TranslationNoteFinder.load_translation_notesc C s: t |ddd��}|�� }W d � n1 s,0 Y |S )Nr& r' r( )r* �read)r" Z21bible_pathr- �22bible_textr# r# r$ r : s &z TranslationNoteFinder.load_biblec C s� g }g }|� � }t|dd�D ]<\}}|| jv rJ|rJd�|�}|�|� g }|�|�� � q |rrd�|�}|�|� td|d �� d d� � �� td|d �� d d� � �� |S )Nr )�start� z,First 10 words of first segmented document: r r z-First 10 words of second segmented document: )r � enumerater �join�append�stripr �split)r" r/ Z documentsZcurrent_documentZ verse_lines�i�lineZjoined_doc_stringr# r# r$ �segment_corpusA s 2324252627 z$TranslationNoteFinder.segment_corpusc C s | � |�}dd� |jD �}|S )Nc S s g | ]}|j D ]28}|j�qqS r# )Zwords�text)�.0ZsentZwordr# r# r$ �29<listcomp>g � z:TranslationNoteFinder.stanza_tokenizer.<locals>.<listcomp>)r Z sentences)r" r: �docZtokensr# r# r$ �stanza_tokenizerc s 30z&TranslationNoteFinder.stanza_tokenizerc C s4 t d� t| jdd�}| �| j�}|�|�}||fS )Nz$Creating tfidf vectorizer and matrix)� � )Z tokenizerZ ngram_range)r r r? r9 r Z
fit_transform)r" r Zsegmented_corpusr r# r# r$ r k s31 32z4TranslationNoteFinder.create_tfidf_vectorizer_matrixc C sv t tj�� ��|�}| j�� }| j| �� }|� � d }t33t ||��}dd� |� � D �}t34t
|� � dd� dd��}|S )Nr c S s i | ]\}}|d kr||�qS )r r# �r; ZfeatureZscorer# r# r$ �35<dictcomp>| r= zATranslationNoteFinder.get_tfidf_book_features.<locals>.<dictcomp>c S s | d S )Nr r# )�itemr# r# r$ �<lambda>~ r= z?TranslationNoteFinder.get_tfidf_book_features.<locals>.<lambda>T)�key�reverse)�list�SRZ36book_codes�keys�indexr Zget_feature_names_outr r! �tolist�dict�zip�items�sorted) r" Z book_codeZ37book_indexZ
feature_namesZdenseZdocument_tfidf_scoresZfeature_scoresZfiltered_feature_scoresZsorted_feature_scoresr# r# r$ �get_tfidf_book_featuress s 38 z-TranslationNoteFinder.get_tfidf_book_featuresc C s� t ddd� |�� D �� �� |d �� }| jd|� d|� d�7 _| jtdd� |�� D �d d39�7 _t d | jd � �� | jd �� S ) Nz0All ngrams in verse guidance is selecting from: c S s g | ]}|�qS r# r# �r; rF r# r# r$ r< � r= z=TranslationNoteFinder.best_ngram_for_note.<locals>.<listcomp>�40greek_termz(What is a good, succinct translation of z from Greek into �?c S s g | ]}|�qS r# r# rR r# r# r$ r< � r= �ngram)�namezBest ngram found for note: )r rJ r5 r r )r" �note� verse_ngramsr rS r# r# r$ �best_ngram_for_note� s $z)TranslationNoteFinder.best_ngram_for_notec s t |�}| j�� |jd }td| j�� d � �� td| j�� d � �� tdddd ��}t�|�}W d � n1 sz0 Y g }td41|� �� |D ]4}td |d � �� |d �� |�� v r�|� |� q�td
dd� |D �� �� | j42�� |jd � td� � �� d}td|� �� |j d } | � | �}43tdt
|44�� �d d� � �� � fdd�|45�� D �} tdt
| �� �d d� � �� g } |D ]X}| �|| |�}
� �� �|
�� �}|t|
� }|d }|d }| � |
||||d�� �q�td | � �� |j |j| d �S ) Nr z)Line 1 of Greek Bible (should be blank): r z0Line 23214 of Greek Bible (should be Matt 1:1): r
ztranslation_notes.jsonr& r' r( z@Let's see if there are any translation notes for this verse: 46 zChecking for existence of: rS z0Greek terms for all translation notes in verse: c S s g | ] }|d �qS )rS r# )r; rW r# r# r$ r< � r= z5TranslationNoteFinder.verse_notes.<locals>.<listcomp>zTarget verse text: ZHindiz Language of target verse text: �bookCodez First 30 n-grams of the book: � c s i | ]\}}|� v r||�qS r# r# rB �Ztarget_verse_textr# r$ rC � r= z5TranslationNoteFinder.verse_notes.<locals>.<dictcomp>z9First five n-grams of the verse along with their scores: � Ztranslation_note)rU � start_pos�end_posrS �47trans_notez Verse notes to be returned: )� verse_ref� line_number�ngrams)rI r r rb r r* r+ r, �lowerr4 r Zstructured_refrQ rH rJ rO rY �find�len)r" ra Zv_refZ
gk_verse_textr- r Ztranslation_notes_in_verserW r rZ Z book_ngramsrX rc rU r^ r_ rS r` r# r\ r$ � verse_notes� sT ( 4849 �� �z!TranslationNoteFinder.verse_notesN)�__name__�50__module__� __qualname__rI Z51verse_onesr r r% r r r9 r? r rQ rY rg r# r# r# r$ r s "r )r+ �pickle�reZ52langdetectr Z sklearn.feature_extraction.textr Zguidancer r r Zopenair Zromanizer r rI r Z TrainingDatar53 r r# r# r# r$ �<module> s