Aluode/PerceptionLabPortable
0
1o
2 .�Yi;$ � @ s� d dl Z d dlZd dlmZmZ d dlmZ d dlmZm Z d dl3Z4d dl m Z d dl
mZ ddlmZ dd lmZ dd5lmZmZ d d lmZmZmZmZ e�e �Z e e� � �Z e!d
d� e D ��Z"eG dd� d��Z#G dd� de�Z$G dd� de�Z%dS )� N)� dataclass�field)�Enum)�Optional�Union)�FileLock)�Dataset� )�$MODEL_FOR_QUESTION_ANSWERING_MAPPING)�PreTrainedTokenizer)�check_torch_load_is_safe�logging� )�
SquadFeatures�SquadV1Processor�SquadV2Processor�"squad_convert_examples_to_featuresc c s � | ]}|j V qd S �N)�6model_type)�.0Zconf� r ��E:\DocsHouse\542 percep lab latest\PerceptionLab\PerceptionLab_Portable\python_embed\Lib\site-packages\transformers/data/datasets/squad.py� <genexpr>" s � r c @ s< e Zd ZU dZedddd�e� id�Zee d< edddid�Z7ee d < ed8dd id�Z e e d < ed9dd
id�Z
e e d< edddid�Ze e d< edddid�Ze e d< edddid�Zee d< edddid�Zee d< edddid�Zee d < ed ddid�Ze e d < ed dd id�Ze e d!< ed"dd#id�Ze e d$< dS )%�SquadDataTrainingArgumentszb10 Arguments pertaining to what data we are going to input our model for training and eval.11 N�helpz!Model type selected in the list: z, )�default�metadatar zFThe input data dir. Should contain the .json files for the SQuAD task.�data_dir� z�The maximum total input sequence length after tokenization. Sequences longer than this will be truncated, sequences shorter will be padded.�max_seq_lengthzVWhen splitting up a long document into chunks, how much stride to take between chunks.�12doc_stride�@ zkThe maximum number of tokens for the question. Questions longer than this will be truncated to this length.�max_query_length� z�The maximum length of an answer that can be generated. This is needed because the start and end predictions are not conditioned on one another.�max_answer_lengthFz1Overwrite the cached training and evaluation sets�overwrite_cachezDIf true, the SQuAD examples contain some that do not have an answer.�version_2_with_negativeg zIIf null_score - best_non_null is greater than the threshold predict null.�null_score_diff_threshold� � n_best_sizer zjlanguage id of input for language-specific xlm models (see tokenization_xlm.PRETRAINED_INIT_CONFIGURATION)�lang_id� z3multiple threads for converting example to features�threads)�__name__�13__module__� __qualname__�__doc__r �join� MODEL_TYPESr �str�__annotations__r r �intr r" r$ r% �boolr& r' �floatr) r* r, r r r r r % sb 14 ���� ��� �� ������ r c @ s e Zd ZdZdZdS )�Split�train�devN)r- r. r/ r9 r: r r r r r8 h s r8 c @ s� e Zd ZU dZeed< ee ed< eed< e ed< dej15dddfded e d16e e
deeef de d e e d efd
d�Zdd� Zdeeejf fdd�ZdS )� SquadDatasetzH17 This will be superseded by a framework-agnostic approach soon.18 �args�features�mode�is_language_sensitiveNF�pt� tokenizer� limit_length� cache_dir�dataset_formatc C s& || _ || _|jr t� nt� | _t|t�r'zt| }W n t y& t d��w || _19|jr/dnd}t j �
|d ur:|n|jd|j� d|jj� d|j� d|� ��} | d }20t|21��� t j �| �r�|js�t�� } t� tj| dd�| _| jd | _| j� d22d �| _ | j� d d �| _ t � d | � d
�t�� | � | j d u s�| j d u r�t �!d| � d�� nX|tj"kr�| j�#|j�| _ n| j�$|j�| _ t%| j ||j|j&|j'|tj(k|j)|d�\| _| _ t�� } t�*| j| j | j d�| � t � d| � dt�� | d�d�� W d � d S W d � d S 1 �s w Y d S )Nz mode is not a valid split nameZv2Zv1Zcached_�_z.lockT)Z weights_onlyr= �dataset�examplesz"Loading features from cached file z [took %.3f s]zDeleting cached file z; will allow dataset and examples to be cached in future run)rG rA r r r" Z is_trainingr, Zreturn_dataset)r= rF rG z!Saving features into cached file z [took z.3fz s])+r< r? r& r r � processor�23isinstancer3 r8 �KeyErrorr> �os�pathr1 r �value� __class__r- r r �existsr% �timer �torch�loadZ old_featuresr= �getrF rG �logger�info�warningr: Zget_dev_examplesZget_train_examplesr r r" r9 r, �save) �selfr< rA rB r> r? rC rD Z version_tagZcached_features_fileZ lock_path�startr r r �__init__w sp 2425 �"�26 �27��28� ����$ zSquadDataset.__init__c C s29 t | j�S r )�lenr= )rX r r r �__len__� s 30zSquadDataset.__len__�returnc C s4 | j | }tj|jtjd�}tj|jtjd�}tj|jtjd�}tj|jtjd�}tj|jtj d�}tj|j31tj d�}|||d�} | j j dv rJ| d= | j j dv rx| �
||d�� | j jrc| �
d|i� | jrx| �
dtj|jtjd�| j j i� | jtjkr�tj|jtjd�}32tj|jtjd�} | �
|33| d �� | S )34N)Zdtype)� input_ids�attention_mask�token_type_ids)�xlmZrobertaZ35distilbertZ camembertr` )Zxlnetra )� cls_index�p_mask�
is_impossibleZlangs)�start_positions�
end_positions)r= rQ Ztensorr^ �longr_ r` rb rc r7 rd r< r �updater& r? Zones�shapeZint64r* r> r8 r9 Zstart_positionZ end_position) rX �i�featurer^ r_ r` rb rc rd Zinputsre rf r r r � __getitem__� s0 36� $ zSquadDataset.__getitem__)r- r. r/ r0 r r4 �listr r8 r6 r9 r r r5 r r3 rZ r\ �dictrQ ZTensorrl r r r r r; m s8 37 ����38���39�L r; )&rK rP � dataclassesr r �enumr �typingr r rQ �filelockr Ztorch.utils.datar Zmodels.auto.modeling_autor40 Ztokenization_utilsr �utilsr r
Zprocessors.squadr r r r Z41get_loggerr- rT rm �keysZMODEL_CONFIG_CLASSES�tupler2 r r8 r; r r r r �<module> s& 42 B