CalebKoster/Translation_Note_Alignment
0
1�
2 ��e � � � d dl mZ d dlZd dlZd dlmZ d dlm Z g d�Z3e4D � cg c] } | dz5 �� c} Z6d7d�Z d8d�Z d9d �Z
yc c} w ) � )�TfidfVectorizerN)�islice)�uroman)A� i� i�10 i i i� ip i� i/ iZ i" iB% i( i�+ i�. i0 i�1 iJ2 ix6 iW@ i�D i=E iHJ i�O i6P i/U i�V iYW i�W i4X iIX iyX i�X iIY i~Y i�Y iwZ i�Z i�^ i�a if iqi i_m io i�p i�q iZr i�r i]s i�s it iDt i�t iu i6u iOu i~v i�v iSw i�w i�w ix ix i.x i�y r c �� � t | dd�� 5 }t |� D ]K \ }}||k( s� |r$t |j � � c cd d d � S |j � c cd d d � S d d d � y # 1 sw Y y xY w)N�r�utf-8��encoding)�open� enumerater �strip)� file_path� line_number�romanize�file�i�lines �:c:\Users\caleb\Bible Translation Project\guidance\tfidf.py�extract_interested_verser P s � �
�i��w� /� (�4� ��� (�G�A�t��K� ��!�$�*�*�,�/�/� (� (� �:�:�<�'�
(� (� (�(� �(� �s �A3� A3�
A3�(A3�3A<c � � g }g }t | dd�� 5 }t |d�� D ]_ \ }}|t v r3|r1dj |� }|r t |� }|j |� g }|j |j
� � �a |r/dj |� }|r t |� }|j |� d d d � |S # 1 sw Y |S xY w)Nr r r11 r )�start� )r r
�verses�joinr �appendr )r r � documents�current_documentr r r �joined_doc_strings r �segment_corpusr \ s� � ��I���
�i��w� /� 0�4� ��Q�/� 2�G�A�t��F�{�#�(+���1A�(B�%� �,2�3D�,E�)� �$�$�%6�7�')�$� � #� #�D�J�J�L� 1� 2� � #���)9� :� ��$*�+<�$=�!� � � � .� /� 0� ��!0� ��s �B!B>�>Cc �p �� t | |�� }t d�� }|j |� }|j � }t �fd�t t � D � � dz12 }t j || j � � j � }t t ||� � } t | �dz13 |�� }14|15rvt d�� } | j |16g� | j � } | D �
ci c] }
|
| j |
d� �� }}
t t! |j# � d� d� � � }|S y17c c}
w ) N)r )� � )� ngram_rangec 3 �4 �K � | ] \ }}|�kD s� |�� � y �w)N� )�.0r �v�interested_lines �r � <genexpr>z*analyze_verse_in_corpus.<locals>.<genexpr>y s �� �� �Q���1�Q� �=P�!�Q�s �
�r r c � � | d S )Nr r& )�items r �<lambda>z)analyze_verse_in_corpus.<locals>.<lambda>� s
� �QU�VW�QX� � T)�key�reversezVerse not found.)r r �
fit_transform�get_feature_names_out�nextr
r �np�array�todense�flatten�dict�zipr �fit�get�sorted�items)r r) r r �tfidf_vectorizer� tfidf_matrix�
feature_names�document_index�scores� scores_dict�interested_verse�tfidf_vectorizer_verse� verse_ngrams�ngram� verse_scores�sorted_verse_scoress ` r �analyze_verse_in_corpusrJ r s% �� � �y�8�<�I�&�6�:��#�1�1�)�<�L�$�:�:�<�M� �Q� �&�(9�Q�Q�TU�U�N� �X�X�l�>�2�:�:�<�
=�
E�
E�
G�F��s�=�&�1�2�K� 0� �?�Q�;N�Ya�b�� �!0�V�!D� � �"�"�$4�#5�6�-�C�C�E� �FR�S�U��{���u�a� 8�8�S� �S� "�6�,�*<�*<�*>�DX�bf�#g� h��"�"�!�� Ts �.D3)F)� sklearn.feature_extraction.textr �pandas�pd�numpyr4 � itertoolsr r r r r r rJ )�xs 0r �<module>rQ sH �� ;� � � � �B18��J � �!�!�A�#� ����,"��K 19 s � ?