Aluode/PerceptionLabPortable
0
1o
2 ��Yi,Q � @ sV d dl mZ d dlmZ d dlmZ ddlmZmZm Z ddl3m Z m Z m
Z
mZmZmZmZmZmZmZmZmZmZmZmZmZmZm Z G dd� d�Z G d d4� d5e �Z G d d � d e �Z G d
d� de �Z G dd� de �Z!G dd� de �Z"G dd� de �Z#G dd� de �Z$G dd� de �Z%G dd� de �Z&edd �d/d"d#��Z'ed$d � &d0d1d,d-��Z(d.S )2� )� annotations)� lru_cache)� getLogger� )� COMMON_SAFE_ASCII_CHARACTERS�TRACE� UNICODE_SECONDARY_RANGE_KEYWORD)�is_accentuated� is_arabic�is_arabic_isolated_form�is_case_variable�is_cjk� is_emoticon� is_hangul� is_hiragana� is_katakana�is_latin�is_punctuation� is_separator� is_symbol�is_thai�is_unprintable�
remove_accent�
unicode_range�is_cjk_uncommonc @ s< e Zd ZdZddd�Zdd d6�Zdd d �Zeddd��ZdS )�MessDetectorPluginzy7 Base abstract class used for mess detection plugins.8 All detectors MUST extend and implement given methods.9 � character�str�return�boolc C � t �)z@10 Determine if given character should be fed in.11 ��NotImplementedError��selfr � r% �E:\DocsHouse\542 percep lab latest\PerceptionLab\PerceptionLab_Portable\python_embed\Lib\site-packages\charset_normalizer/md.py�eligible' � zMessDetectorPlugin.eligible�Nonec C r )z�12 The main routine to be executed upon character.13 Insert the logic in witch the text would be considered chaotic.14 r! r# r% r% r& �feed- s zMessDetectorPlugin.feedc C r )zB15 Permit to reset the plugin to the initial state.16 r! �r$ r% r% r& �reset4 r( zMessDetectorPlugin.reset�floatc C r )z�17 Compute the chaos ratio based on what your feed() has seen.18 Must NOT be lower than 0.; No restriction gt 0.19 r! r+ r% r% r& �ratio: s zMessDetectorPlugin.ratioN�r r r r �r r r r) �r r) �r r- ) �__name__�20__module__� __qualname__�__doc__r' r* r, �propertyr. r% r% r% r& r ! s 212223r c @ �B e Zd Zddd�Zddd �Zdd24d �Zdd d
�Zeddd��ZdS )� TooManySymbolOrPunctuationPluginr r) c C s" d| _ d| _d| _d | _d| _d S )Nr F)�_punctuation_count�
_symbol_count�_character_count�_last_printable_charZ_frenzy_symbol_in_wordr+ r% r% r& �__init__D s25 26z)TooManySymbolOrPunctuationPlugin.__init__r r r c C � |� � S �N�� isprintabler# r% r% r& r' L � z)TooManySymbolOrPunctuationPlugin.eligiblec C sp | j d7 _ || jkr3|tvr3t|�r | jd7 _n|�� du r3t|�r3t|�du r3| jd7 _|| _d S )Nr F� ) r< r= r r r: �isdigitr r r; r# r% r% r& r* O s 27 � 28z%TooManySymbolOrPunctuationPlugin.feedc C s d| _ d| _d| _d S �Nr )r: r< r; r+ r% r% r& r, a � 29z&TooManySymbolOrPunctuationPlugin.resetr- c C s0 | j dkrdS | j| j | j }|dkr|S dS )Nr � �333333�?)r< r: r; )r$ Zratio_of_punctuationr% r% r& r. f s 3031�z&TooManySymbolOrPunctuationPlugin.ratioNr1 r/ r0 r2 � r3 r4 r5 r> r' r* r, r7 r. r% r% r% r& r9 C s 32333435r9 c @ r8 )�TooManyAccentuatedPluginr r) c C � d| _ d| _d S rF �r< �_accentuated_countr+ r% r% r& r> s � 36z!TooManyAccentuatedPlugin.__init__r r r c C r? r@ )�isalphar# r% r% r& r' w rC z!TooManyAccentuatedPlugin.eligiblec C �, | j d7 _ t|�r| jd7 _d S d S �Nr )r< r rN r# r% r% r& r* z � �z TooManyAccentuatedPlugin.feedc C rL rF rM r+ r% r% r& r, � rO z TooManyAccentuatedPlugin.resetr- c C s* | j dk rdS | j| j }|dkr|S dS )N� rH gffffff�?rM )r$ Zratio_of_accentuationr% r% r& r. � s 37 z TooManyAccentuatedPlugin.ratioNr1 r/ r0 r2 rJ r% r% r% r& rK r s 38394041rK c @ r8 )�UnprintablePluginr r) c C rL rF )�_unprintable_countr< r+ r% r% r& r> � rO zUnprintablePlugin.__init__r r r c C � dS �NTr% r# r% r% r& r' � � zUnprintablePlugin.eligiblec C s( t |�r | jd7 _| jd7 _d S rR )r rV r< r# r% r% r& r* � s zUnprintablePlugin.feedc C s42 d| _ d S rF )rV r+ r% r% r& r, � s 43zUnprintablePlugin.resetr- c C � | j dkrdS | jd | j S )Nr rH rT )r< rV r+ r% r% r& r. � � 44zUnprintablePlugin.ratioNr1 r/ r0 r2 rJ r% r% r% r& rU � s 45464748rU c @ r8 )� SuspiciousDuplicateAccentPluginr r) c C � d| _ d| _d | _d S rF ��_successive_countr< �_last_latin_characterr+ r% r% r& r> � s 49z(SuspiciousDuplicateAccentPlugin.__init__r r r c C s |� � ot|�S r@ )rP r r# r% r% r& r' � s z(SuspiciousDuplicateAccentPlugin.eligiblec C st | j d7 _ | jd ur5t|�r5t| j�r5|�� r%| j�� r%| jd7 _t|�t| j�kr5| jd7 _|| _d S rR )r< r` r �isupperr_ r r# r% r% r& r* � s 50��51z$SuspiciousDuplicateAccentPlugin.feedc C r] rF r^ r+ r% r% r& r, � rG z%SuspiciousDuplicateAccentPlugin.resetr- c C rZ )Nr rH rD )r< r_ r+ r% r% r& r. � r[ z%SuspiciousDuplicateAccentPlugin.ratioNr1 r/ r0 r2 rJ r% r% r% r& r\ � s 52535455r\ c @ r8 )�SuspiciousRanger r) c C r] rF )�"_suspicious_successive_range_countr< �_last_printable_seenr+ r% r% r& r> � rG zSuspiciousRange.__init__r r r c C r? r@ rA r# r% r% r& r' � rC zSuspiciousRange.eligiblec C sx | j d7 _ |�� st|�s|tv rd | _d S | jd u r"|| _d S t| j�}t|�}t||�r7| jd7 _|| _d S rR )r< �isspacer r rd r � is_suspiciously_successive_rangerc )r$ r �unicode_range_a�unicode_range_br% r% r& r* � s ��56575859zSuspiciousRange.feedc C r] rF )r< rc rd r+ r% r% r& r, � rG zSuspiciousRange.resetr- c C s" | j dkrdS | jd | j }|S )N�
rH rD )r< rc )r$ Z ratio_of_suspicious_range_usager% r% r& r. � s 60�zSuspiciousRange.ratioNr1 r/ r0 r2 rJ r% r% r% r& rb � s 61626364rb c @ r8 )�SuperWeirdWordPluginr r) c C s@ d| _ d| _d| _d| _d| _d| _d| _d| _d| _d| _ d S )Nr F� )65� _word_count�_bad_word_count�_foreign_long_count�_is_current_word_bad�_foreign_long_watchr< �_bad_character_count�_buffer�_buffer_accent_count�_buffer_glyph_countr+ r% r% r& r> � s 66z SuperWeirdWordPlugin.__init__r r r c C rW rX r% r# r% r% r& r' rY z SuperWeirdWordPlugin.eligiblec C s� |� � rc| j|7 _t|�r| jd7 _| jdu rFt|�du s%t|�rFt|�du rFt|�du rFt|�du rFt |�du rFt67|�du rFd| _t|�sZt|�sZt|�sZt |�sZt68|�ra| j d7 _ d S | jshd S |� � sut
|�sut|��r$| j�r$| jd7 _t| j�}| j|7 _|dkr�| j| dkr�d| _n4t| jd �r�| jd �� r�tdd� | jD ��du r�| jd7 _d| _n| j dkr�d| _| jd7 _|d kr�| jr�d69d � t| jtd |��D �}d}|r�t|�| d
kr�d}|s�| jd7 _d| _| j�r| jd7 _| jt| j�7 _d| _d| _d| _d | _d | _ d S |dv�rA|�� du �rCt|��rEd| _| j|7 _d S d S d S d S )Nr FT� � �?�����c s s � | ]}|� � V qd S r@ �ra )�.0�_r% r% r& � <genexpr>8 s � z,SuperWeirdWordPlugin.feed.<locals>.<genexpr>� c S s g | ]70\}}|� � r|�qS r% rx )ry �c�ir% r% r& �71<listcomp>@ s ��z-SuperWeirdWordPlugin.feed.<locals>.<listcomp>r rI rk > �|rz �=�<�~�>�-) rP rr r rs rp r r
r r r r rt re r r rl �lenr< ro ra �allrn �zip�rangerm rq rE r )r$ r Z
buffer_lengthZcamel_case_dstZprobable_camel_casedr% r% r& r* s� 72 ���������73 � �74�7576��zSuperWeirdWordPlugin.feedc C s4 d| _ d| _d| _d| _d| _d| _d| _d| _d S )Nrk Fr )rr ro rp rm rl r< rq rn r+ r% r% r& r, _ s 77zSuperWeirdWordPlugin.resetr- c C s$ | j dkr | jdkr dS | j| j S )N�78 r rH )rl rn rq r< r+ r% r% r& r. i s zSuperWeirdWordPlugin.ratioNr1 r/ r0 r2 rJ r% r% r% r& rj � s 79808182Q83rj c @ sF e Zd ZdZddd�Zdd d84�Zdd d �Zdd
d�Zeddd��Z dS )�CjkUncommonPluginz<85 Detect messy CJK text that probably means nothing.86 r r) c C rL rF �r< �_uncommon_countr+ r% r% r& r> v rO zCjkUncommonPlugin.__init__r r r c C � t |�S r@ )r
r# r% r% r& r' z rC zCjkUncommonPlugin.eligiblec C rQ rR )r< r r� r# r% r% r& r* } s87 �zCjkUncommonPlugin.feedc C rL rF r� r+ r% r% r& r, � rO zCjkUncommonPlugin.resetr- c C s. | j dk rdS | j| j }|dkr|d S dS )NrT rH rv r� r� )r$ Zuncommon_form_usager% r% r& r. � s 88 zCjkUncommonPlugin.ratioNr1 r/ r0 r2 )89r3 r4 r5 r6 r> r' r* r, r7 r. r% r% r% r& r� q s 90919293r� c @ r8 )�ArchaicUpperLowerPluginr r) c C s. d| _ d| _d| _d| _d| _d | _d| _d S )NFr T)�_buf� _character_count_since_last_sep� _successive_upper_lower_count�#_successive_upper_lower_count_finalr< �_last_alpha_seen�_current_ascii_onlyr+ r% r% r& r> � s 94z ArchaicUpperLowerPlugin.__init__r r r c C rW rX r% r# r% r% r& r' � rY z ArchaicUpperLowerPlugin.eligiblec C s$ |� � ot|�}|du }|rC| jdkrC| jdkr+|�� du r+| jdu r+| j| j7 _d| _d| _d | _d| _| j d7 _ d| _d S | jdu rQ|�95� du rQd| _| jd ur|� � r_| j� � sh|� � r|| j� � r|| jdu rx| jd7 _d| _nd| _nd| _| j d7 _ | jd7 _|| _d S )NFr �@ r TrD )
rP r r� rE r� r� r� r� r� r< �isasciira �islower)r$ r Z is_concernedZ chunk_sepr% r% r& r* � s@ 96 97�98��99100z ArchaicUpperLowerPlugin.feedc C s. d| _ d| _d| _d| _d | _d| _d| _d S )Nr FT)r< r� r� r� r� r� r� r+ r% r% r& r, � s 101z ArchaicUpperLowerPlugin.resetr- c C s | j dkrdS | j| j S )Nr rH )r< r� r+ r% r% r& r. � s 102 z ArchaicUpperLowerPlugin.ratioNr1 r/ r0 r2 rJ r% r% r% r& r� � s 103104
105106* r� c @ sB e Zd Zddd�Zddd�Zdd107d �Zdd d
�Zeddd��ZdS )�ArabicIsolatedFormPluginr r) c C rL rF �r< �_isolated_form_countr+ r% r% r& r> � rO z!ArabicIsolatedFormPlugin.__init__c C rL rF r� r+ r% r% r& r, � rO z ArabicIsolatedFormPlugin.resetr r r c C r� r@ )r108 r# r% r% r& r' � rC z!ArabicIsolatedFormPlugin.eligiblec C rQ rR )r< r r� r# r% r% r& r* � rS z ArabicIsolatedFormPlugin.feedr- c C s | j dk rdS | j| j }|S )NrT rH r� )r$ Zisolated_form_usager% r% r& r. � s 109 z ArabicIsolatedFormPlugin.ratioNr1 r/ r0 r2 ) r3 r4 r5 r> r, r' r* r7 r. r% r% r% r& r� � s 110111112113r� � )�maxsizerg �114str | Nonerh r r c C sv | du s|du r115dS | |krdS d| v rd|v rdS d| v s"d|v r$dS d| v s,d|v r6d| v s4d|v r6dS | � d�|� d�}}|D ]}|tv rJqC||v rQ dS qC| dv |dv }}|s_|rid | v sgd |v ridS |ro|rodS d116| v swd117|v r�d | v sd |v r�dS | d ks�|d kr�dS d | v s�d |v s�| d v r�|d v r�d
| v s�d
|v r�dS d| v s�d|v r�dS | d ks�|d kr�dS dS )za118 Determine if two Unicode range seen next to each other can be considered as suspicious.119 NTFZLatinZ EmoticonsZ Combining� )�Hiragana�KatakanaZCJKZHangulz Basic Latin)r� r� � PunctuationZForms)�splitr )rg rh Zkeywords_range_aZkeywords_range_b�elZrange_a_jp_charsZrange_b_jp_charsr% r% r& rf � sZ ����rf i 皙�����?F�decoded_sequencer �maximum_thresholdr- �debugc C sR dd� t �� D �}t| �d }d}|dk rd}n |dkr d}nd }t| d120 t|��D ]2\}}|D ] } | �|�r<| �|� q0|d krG|| d ksM||d kr\td d
� |D ��}||kr\ nq*|r�td�}121|122� t123d|� d|� d|� �� t| �dkr�|124� t125d| dd� � �� |126� t127d| dd� � �� |D ]} |128� t129| j � d| j � �� q�t
|d�S )zw130 Compute a mess ratio given a decoded bytes sequence. The maximum threshold does stop the computation earlier.131 c S s g | ]}|� �qS r% r% )ry Zmd_classr% r% r& r N s �z mess_ratio.<locals>.<listcomp>r rH i � r� r� � �132r c s s � | ]}|j V qd S r@ )r. )ry �dtr% r% r& r{ e s � z mess_ratio.<locals>.<genexpr>Zcharset_normalizerzIMess-detector extended-analysis start. intermediary_mean_mess_ratio_calc=z mean_mess_ratio=z maximum_threshold=� zStarting with: Nz
Ending with: i����z: � )r �__subclasses__r� r� r� r' r* �sumr �logr � __class__r. �round) r� r� r� Z detectors�lengthZmean_mess_ratioZ!intermediary_mean_mess_ratio_calcr �indexZdetector�loggerr� r% r% r& �133mess_ratioF sN � 134135� ����� 136r� N)rg r� rh r� r r )r� F)r� r r� r- r� r r r- ))�137__future__r � functoolsr �loggingr Zconstantr r r �utilsr r138 r r r
r r r r r r r r r r r r r r r9 rK rU r\ rb rj r� r� r� rf r� r% r% r% r& �<module> s( P"/%1v#L I�