Aluode/PerceptionLabPortable
0
1o
2 ��Yi�1 � @ s� d dl mZ d dlmZ d dlmZ d dlmZ d dlm Z d dl3m Z m Z m
Z
mZ ddlmZmZ dd lmZmZmZ G d4d � d �ZG d d
� d
�Zeeef Ze
e ZG dd� d�Z dS )� )� annotations)�aliases)�sha256)�dumps)�sub)�Any�Iterator�List�Tuple� )� RE_POSSIBLE_ENCODING_INDICATION�TOO_BIG_SEQUENCE)� iana_name�is_multi_byte_encoding�
unicode_rangec @ sH e Zd Z dGdHdd�ZdIdd�ZdIdd�ZedJdd��ZdKdd�ZdKd d �Z dLd d �Z5edKd!d"��Z edMd$d%��Z edNd&d'��Z
edNd(d)��ZedMd*d+��ZedKd,d-��ZedJd.d/��ZedJd0d1��ZedJd2d3��ZedJd4d5��ZedOd6d7��ZedPd9d:��ZedNd;d<��ZedMd=d>��ZedMd?d@��ZdQdRdCdD�ZedKdEdF��ZdS )S� CharsetMatchN�payload�bytes�guessed_encoding�str�mean_mess_ratio�float�has_sig_or_bom�bool� languages�CoherenceMatches�decoded_payload�6str | None�preemptive_declarationc C sL || _ || _|| _|| _|| _d | _g | _d| _d | _d | _ || _7|| _ d S )N� ) �_payload� _encoding�_mean_mess_ratio�8_languages�_has_sig_or_bom�_unicode_ranges�_leavesZ_mean_coherence_ratio�_output_payload�_output_encoding�_string�_preemptive_declaration)�selfr r r r r r r � r, ��E:\DocsHouse\542 percep lab latest\PerceptionLab\PerceptionLab_Portable\python_embed\Lib\site-packages\charset_normalizer/models.py�__init__ s 910zCharsetMatch.__init__�other�object�returnc C s> t |t�st |t�rt|�| jkS dS | j|jko | j|jkS )NF)�11isinstancer r r �encoding� fingerprint�r+ r/ r, r, r- �__eq__* s12 1314zCharsetMatch.__eq__c C s� t |t�st�t| j|j �}t| j|j �}|dk r%|dkr%| j|jkS |dk r@|dkr@t| j�tkr:| j|jk S | j |j kS | j|jk S )zQ15 Implemented to make sorted available upon CharsetMatches items.16 g{�G�z�?g{�G�z�?)17r2 r �18ValueError�abs�chaos� coherence�lenr r
�multi_byte_usage)r+ r/ Zchaos_differenceZcoherence_differencer, r, r- �__lt__1 s 19 zCharsetMatch.__lt__c C s dt t| ��t | j� S )Ng �?)r; r �raw�r+ r, r, r- r< G s z CharsetMatch.multi_byte_usagec C s" | j d u rt| j| jd�| _ | j S )N�strict)r) r r r! r? r, r, r- �__str__K s 20zCharsetMatch.__str__c C s d| j � d| j� d�S )Nz<CharsetMatch 'z' bytes(z)>)r3 r4 r? r, r, r- �__repr__Q s zCharsetMatch.__repr__�Nonec C s8 t |t�r || krtd�|j���d |_| j�|� d S )Nz;Unable to add instance <{}> as a submatch of a CharsetMatch)r2 r r7 �format� __class__r) r& �appendr5 r, r, r- � add_submatchT s ��zCharsetMatch.add_submatchc C � | j S �N)r! r? r, r, r- r3 _ � zCharsetMatch.encoding� list[str]c C sD g }t �� D ]\}}| j|kr|�|� q| j|kr |�|� q|S )z�21 Encoding name are known by many name, using this could help when searching for IBM855 when it's listed as CP855.22 )r �itemsr3 rF )r+ Z
also_known_as�u�pr, r, r- �encoding_aliasesc s 23 2425�z CharsetMatch.encoding_aliasesc C rH rI �r$ r? r, r, r- �bomp rJ zCharsetMatch.bomc C rH rI rP r? r, r, r- �byte_order_markt rJ z CharsetMatch.byte_order_markc C s dd� | j D �S )z�26 Return the complete list of possible languages found in decoded sequence.27 Usually not really useful. Returned list may be empty even if 'language' property return something != 'Unknown'.28 c S s g | ]}|d �qS )r r, )�.0�er, r, r- �29<listcomp>~ � z*CharsetMatch.languages.<locals>.<listcomp>�r# r? r, r, r- r x s zCharsetMatch.languagesc C sp | j s1d| jv r30dS ddlm}m} t| j�r || j�n|| j�}t|�dks+d|v r-dS |d S | j d d S )z�31 Most probable language found in decoded sequence. If none were detected or inferred, the property will return32 "Unknown".33 �asciiZEnglishr )�encoding_languages�mb_encoding_languagesz Latin Based�Unknown)r# �could_be_from_charsetZcharset_normalizer.cdrY rZ r r3 r; )r+ rY rZ r r, r, r- �language� s 34 ��zCharsetMatch.languagec C rH rI )r" r? r, r, r- r9 � rJ zCharsetMatch.chaosc C s | j sdS | j d d S )Nr r r rW r? r, r, r- r: � s zCharsetMatch.coherencec C � t | jd dd�S �N�d � )�ndigits)�roundr9 r? r, r, r- �
percent_chaos� � zCharsetMatch.percent_chaosc C r^ r_ )rc r: r? r, r, r- �percent_coherence� re z CharsetMatch.percent_coherencec C rH )z+35 Original untouched bytes.36 )r r? r, r, r- r> � s zCharsetMatch.raw�list[CharsetMatch]c C rH rI )r& r? r, r, r- �submatch� rJ zCharsetMatch.submatchc C � t | j�dkS �Nr )r; r& r? r, r, r- � has_submatch� s zCharsetMatch.has_submatchc C s@ | j d ur| j S dd� t| �D �}ttdd� |D ���| _ | j S )Nc S s g | ]}t |��qS r, )r )rS �charr, r, r- rU � rV z*CharsetMatch.alphabets.<locals>.<listcomp>c S s h | ]}|r|�qS r, r, )rS �rr, r, r- � <setcomp>� rV z)CharsetMatch.alphabets.<locals>.<setcomp>)r% r �sorted�list)r+ Zdetected_rangesr, r, r- � alphabets� s37 38zCharsetMatch.alphabetsc C s | j gdd� | jD � S )z�39 The complete list of encoding that output the exact SAME str result and therefore could be the originating40 encoding.41 This list does include the encoding available in property 'encoding'.42 c S s g | ]}|j �qS r, )r3 )rS �mr, r, r- rU � s z6CharsetMatch.could_be_from_charset.<locals>.<listcomp>)r! r& r? r, r, r- r\ � s z"CharsetMatch.could_be_from_charset�utf_8r3 c s~ � j du s43� j |kr<|� _ t� �}� jdur5� j�� dvr5tt� fdd�|dd� dd�}||dd� }|�|d�� _� jS ) z�44 Method to get re-encoded bytes payload using given target encoding. Default to UTF-8.45 Any errors will be simply ignored by the encoder NOT replaced.46 N)zutf-8�utf8rs c s< | j | �� d | �� d � �| �� d t� j��dd��S )Nr r �_�-)�string�span�replace�groupsr r( )rr r? r, r- �<lambda>� s 47�z%CharsetMatch.output.<locals>.<lambda>i r )�country )r( r r* �lowerr r �encoder' )r+ r3 �decoded_stringZpatched_headerr, r? r- �output� s 48�4950�51zCharsetMatch.outputc C s t | �� ��� S )zw52 Retrieve the unique SHA256 computed using the transformed (re-encoded) payload. Not the original one.53 )r r� � hexdigestr? r, r, r- r4 � s zCharsetMatch.fingerprint)NN)r r r r r r r r r r r r r r )r/ r0 r1 r )r1 r �r1 r )r/ r r1 rC )r1 rK �r1 r )r1 r )r1 rg )rs )r3 r r1 r ) �__name__�54__module__� __qualname__r. r6 r= �propertyr< rA rB rG r3 rO rQ rR r r] r9 r: rd rf r> rh rk rq r\ r� r4 r, r, r, r- r
sX �55 56 575859 r c @ sb e Zd ZdZd d dd�Zd d d60�Zd!dd�Zd"dd�Zd#dd�Zd$dd�Z d%dd�Z61d%d d �Z dS )&�CharsetMatchesz�62 Container with every CharsetMatch items ordered by default from most probable to the less one.63 Act like a list(iterable) but does not implements all related methods.64 N�results�list[CharsetMatch] | Nonec C s |r t |�| _d S g | _d S rI )ro �_results)r+ r� r, r, r- r. � s zCharsetMatches.__init__r1 �Iterator[CharsetMatch]c c s � | j E d H d S rI �r� r? r, r, r- �__iter__� s �zCharsetMatches.__iter__�item� int | strr c C sJ t |t�r65| j| S t |t�r#t|d�}| jD ] }||jv r"| S qt�)z�66 Retrieve a single item either by its position or encoding name (alias may be used here).67 Raise KeyError upon invalid index or encoding not present in results.68 F)r2 �intr� r r r\ �KeyError)r+ r� �resultr, r, r- � __getitem__ s 697071727374�zCharsetMatches.__getitem__r� c C s75 t | j�S rI �r; r� r? r, r, r- �__len__ s 76zCharsetMatches.__len__r c C ri rj r� r? r, r, r- �__bool__ s zCharsetMatches.__bool__rC c C s| t |t�std�t|j����t|j�tk r0| j D ]}|j77|j78kr/|j |j kr/|� |� dS q| j �
|� t| j �| _ dS )z~79 Insert a single match. Will be inserted accordingly to preserve sort.80 Can be inserted as a submatch.81 z-Cannot append instance '{}' to CharsetMatchesN)r2 r r7 rD r rE r; r> r
r� r4 r9 rG rF ro )r+ r� �matchr, r, r- rF s 82��8384� zCharsetMatches.append�CharsetMatch | Nonec C s | j sdS | j d S )zQ85 Simply return the first match. Strict equivalent to matches[0].86 Nr r� r? r, r, r- �best) s 87zCharsetMatches.bestc C s | � � S )zP88 Redundant method, call the method best(). Kept for BC reasons.89 )r� r? r, r, r- �first1 s zCharsetMatches.firstrI )r� r� )r1 r� )r� r� r1 r )r1 r� r� )r� r r1 rC )r1 r� ) r� r� r� �__doc__r. r� r� r� r� rF r� r� r, r, r, r- r� � s 909192939495r� c @ s. e Zd Zddd�Zeddd��Zd dd�ZdS ) �CliDetectionResult�pathr r3 r rO rK �alternative_encodingsr] rq r r r9 r r: � unicode_path� is_preferredc C sF || _ |96| _|| _|| _|| _|| _|| _|| _|| _| | _ | | _97d S rI ) r� r� r3 rO r� r] rq r r9 r: r� ) r+ r� r3 rO r� r] rq r r9 r: r� r� r, r, r- r. = s 98zCliDetectionResult.__init__r1 �dict[str, Any]c C s2 | j | j| j| j| j| j| j| j| j| j | j99d� S )N� r� r3 rO r� r] rq r r9 r: r� r� r� r? r, r, r- �__dict__W s �zCliDetectionResult.__dict__c C s t | jddd�S )NT� )� ensure_ascii�indent)r r� r? r, r, r- �to_jsong s zCliDetectionResult.to_jsonN)r� r r3 r rO rK r� rK r] r rq rK r r r9 r r: r r� r r� r )r1 r� r� )r� r� r� r. r� r� r� r, r, r, r- r� < s100 101 r� N) �102__future__r Zencodings.aliasesr �hashlibr �jsonr �rer �typingr r r r103 Zconstantr r
�utilsr r r r r� r r ZCoherenceMatchr r� r, r, r, r- �<module> s i C