Aluode/PerceptionLabPortable
0
1o
2 ��YiC� � @ s d Z ddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl3Z4ddl Z ddl m
Z
ddlZddlmZmZ ddlmZ ddlmZmZmZmZmZmZmZm Z m Z m Z m Z m Z ddl!m"Z"m#Z# ddl$m%Z%m&Z&m'Z' dd l(m)Z)m*Z*m+Z+ d�d d �Z,d
d� Z-dd� Z.dd� Z/dd� Z0e)dddg�dd� �Z1dd� Z2d d � Z3d d � Z4d d!� Z5d"d#� Z6d$d%� Z7d&d'� Z8d(d)� Z9g fd*d+�Z:d,d-� Z;e'e)d.dd/g�d0d1� ��Z<e'd2d3� �Z=d4d5� Z>d6d7� Z?d8d9� Z@d:d;� ZAe)d<d=gd>d/fg d?dfg�d@dA� �ZBdBdC� ZCdDdE� ZDejE�)dFddg�dGdH� �ZFdIdJ� ZGdKdL� ZHdMdN� ZIdOdP� ZJdQdR� ZKdSdT� ZLdUdV� ZMdWdX� ZNdYdZ� ZOd[d\� ZPd]d^d_�d`da�ZQdbeRdceSddeSfdedf�ZTdgdh� ZUdidj� ZVd�dkdl�ZWdmdn� ZXdodp� ZYdqdr� ZZdsdt� Z[dudv� Z\dwdx� Z]dydz� Z^d�d{d|�Z_e&d}d~� �Z`e&dd�� �Zad�d�� ZbG d�d�� d�e�ZcG d�d�� d�e�Zde)d�dee� ef� g�d�d�� �Zgd�d�� Zhd�d�� Zid�d�� Zjd�d�� Zkd�d�� Zld�d�� Zmd�d�� Znd�d�� Zod�d�� Zpd�d�d��ZqejE�)d�d�d�d��d/d�d�d�id��g�d�d�� �Zrd�d�� ZsG d�d�� d��ZtG d�d�� d��ZuG d�d�� d��ZvdS )�z5Test the memory module.6� N)�Path)�FileSystemStoreBackend�StoreBackendBase)�hash) �_FUNCTION_HASHES�_STORE_BACKENDS�JobLibCollisionWarning�
MemorizedFunc�MemorizedResult�Memory�NotMemorizedFunc�NotMemorizedResult�_build_func_identifier�_store_backend_factory�
expires_after�register_store_backend)�Parallel�delayed)�np�with_multiprocessing�7with_numpy)� parametrize�raises�warns� c C s | d | S )z-A module-level function for testing purposes.� � ��x�yr r ��E:\DocsHouse\542 percep lab latest\PerceptionLab\PerceptionLab_Portable\python_embed\Lib\site-packages\joblib/test/test_memory.py�f/ s r! c C sZ t |dd�}|�| �} td�D ]}td�D ]}| |�|ks J �t|�|d ks)J �qqdS )z�Given a function and an accumulator (a list that grows every8 time the function is called), check that the function can be9 decorated by memory to be a lazy identity.10 r ��location�verbose� r r N)r �cache�range�len)�func� accumulatorr# �memory�i�_r r r �check_identity_lazy6 s 11 ��r. c C sV | j �� \}tj�|jd�}t|d��}|�d� W d � d S 1 s$w Y d S )N�12output.pkl�w�garbage)�
store_backend� get_items�os�path�join�open�write)r+ Zsingle_cache_itemZoutput_filenamer! r r r �corrupt_single_cache_itemE s13 "�r9 c s" g � � fdd�}|� | d|� � S )Nc s � � | � d S �N��append)�item�Zrecordedr r �append_to_recordS s z6monkeypatch_cached_func_warn.<locals>.append_to_record�warn)�setattr)r) Zmonkeypatch_fixturer? r r> r � monkeypatch_cached_func_warnL s rB c s� t � � � fdd�}t|� | j� dD ]M}dD ]*}t| jd||d�}tj| jdd� |�|�}|d � |jd14d � t� �}|d �}qt� �|d ksKJ �|� |d �|ksUJ �t� �|d ks_J �qd |_15t| jd
d�}|�|�d � dS )z&Simple test of memory lazy evaluation.c � � � d� | S �Nr r; ��arg�r* r r r! c � 16z"test_memory_integration.<locals>.f)FT)�rN�17 )r# r$ � mmap_mode�compressT��
ignore_errorsr F)r@ �__main__r r" N) �listr. �strpathr �shutil�rmtreer& �clearr( �eval�18__module__)�tmpdirr! rL rK r+ �g�current_accumulator�outr rG r �test_memory_integration\ s. � 19 20r[ �call_before_reducingTFc s dD ]�}d}d� |�}i }ttt�|�|dd�d |� |d }d|_|d�dks*J �|jj|ks2J �t| j d d21�}|�22|�� t t �
| d ��dksKJ �t �
| d �d }d |v sZJ �| d | } |d kr�t �
| �dgksnJ �t �
| d �g ksyJ �|r�� d
� t t �
| d ��dks�J �tdd�� fdd�dD �� n8tdd�� fdd�dD �� td�D ]}23t t �
| d ��d
kr� nt�d� q�t t �
| d ��d
ks�J �� d
� t t �
| d ��dks�J �qt t �
| d ��dks�J �� d
� t t �
| d ��dks�J �qd S )N)r r z024 def f(x):25 return x26 z <ipython-input-{}-000000000000>�exec)�filename�moder! rO r r r" �joblibz
ipython-inputr% r �Zn_jobsc 3 � � | ] }t � �|�V qd S r: �r ��.0r, �Zcached_fr r � <genexpr>� � � zHtest_parallel_call_cached_function_defined_in_jupyter.<locals>.<genexpr>)r r c 3 rb r: rc rd rf r r rg � rh � 皙�����?� )�formatr] �compile�textwrap�dedentrV �__code__� co_filenamer rQ r& r( r4 �listdirr r' �time�sleep) rW r\ Z27session_noZipython_cell_sourceZipython_cell_idZ my_localsr! r+ Zf_cache_relative_directoryZf_cache_directoryr- r rf r �5test_parallel_call_cached_function_defined_in_jupyter� sT 28�� 29 �ru c s^ t � � � fdd�} tddd�}|�| �}td�D ]}t� �}|d� t� �|d ks,J �qdS )z*Test memory with location=None: no memoizec rC rD r; rE rG r r �ff� rH ztest_no_memory.<locals>.ffNr r" rk r )rP r r& r' r( )rv r+ Zggr- rY r rG r �test_no_memory� s 30 �rw c sR t � � d31� fdd� }t|� | j� t| jdd�}|�|�}|ddd �dks'J �dS ) z3Test memory with a function with keyword arguments.Nr c rC rD r; �Zarg1Zarg2rG r r rX rH z test_memory_kwarg.<locals>.gr r" � r rx rD )rP r. rQ r r& )rW rX r+ r rG r �test_memory_kwarg s 32rz c s, t � � � fdd��t�fdd�� | j� dS )z*Test memory with a function with a lambda.c rC �z*A helper function to define l as a lambda.r r; �r rG r r �helper � 33z"test_memory_lambda.<locals>.helperc s � | �S r: r r| )r} r r �<lambda> � z$test_memory_lambda.<locals>.<lambda>N)rP r. rQ )rW r )r* r} r �test_memory_lambda s r� c C s� t | jdd�}|jdd� �}|}|jdd� �}|}tt��}|d� |d� W d � n1 s0w Y t|�dks=J �dt|d j�v sHJ �dS ) z=Check that name collisions with functions will raise warningsr r" c S � | S )z&A first function called name_collisionr r| r r r �name_collision$ � z2test_memory_name_collision.<locals>.name_collisionc S r� )z'A second function called name_collisionr r| r r r r� + r� r N� collision)r rQ r& r r r( �str�message)rW r+ r� �a�b�warninfor r r �test_memory_name_collision s 34353637 �r� c C s� t | jdd�}|�dd� �}|�dd� �}tt�� }|d�dks"J �|d�dks*J �|d�dks2J �W d � n1 s<w Y t|�dksIJ �d S ) Nr r" c S r� r: r r| r r r r = � z7test_memory_warning_lambda_collisions.<locals>.<lambda>c S � | d S rD r r| r r r r > r� r r rk )r rQ r& r r r( )rW r+ r� r� r� r r r �%test_memory_warning_lambda_collisions: s 38 �r� c C s� t | jdd�}td�}|�|�}td�}|�|�}tt��}|d� |d� |d� W d � n1 s4w Y t|�dksAJ �dt|d j�� � v sNJ �d S )Nr r" � lambda x: xz
lambda x: x+1r r z
cannot detect)39r rQ rU r& r r r( r� r� �lower)rW r+ Za1�b1r� r r r �'test_memory_warning_collision_detectionJ s 40414243 � r� c s8 t � � � fdd�}ddl}|�|d�}t|� | j� dS )z#Test memory with functools.partial.c s � � d� |S r{ r; r rG r r r) ` r~ z!test_memory_partial.<locals>.funcr Nr )rP � functools�partialr. rQ )rW r) r� �functionr rG r �test_memory_partial\ s44 r� c C s4 t | jdd�}td�}|�|�}|d�dksJ �dS )zESmoke test memory with a function with a function defined in an eval.r r" r� r N)r rQ rU r& )rW r+ �m�mmr r r �test_memory_evall s 45r� c C s t | �}| �d� |S )zsA function with a side effect in its arguments.46 47 Return the length of its argument and append one element.48 N)r( r< )r Zlen_xr r r �count_and_appendv s 49r� c C s8 t | jdd�}|�t�}|� dksJ �|� dksJ �dS )zsCheck that if a function has a side effect in its arguments, it50 should use the hash of changing arguments.51 r r" r N)r rQ r& r� )rW r+ r) r r r �test_argument_change� s 52r� rK rI c s� t � � d � fdd� }t| j|dd�}|�|�}tj�d�}td�D ]#}|�d�}td�D ]}t� ||�|k�s9J �t53� �|d ksCJ �q,q!dS )54z.Test memory with a function with numpy arrays.Nc rC rD r; rE rG r r �n� rH z test_memory_numpy.<locals>.nr �r# rK r$ r% )rJ rJ r r: ) rP r rQ r& r �randomZ RandomStater' Z
random_sample�allr( ) rW rK r� r+ Zcached_nZrndr, r� r- r rG r �test_memory_numpy� s 55 56 ��r� c 57 C s� t | jddd�}|�� dd� �}t�d�}||�}||�}t|tj�s%J �|jdks,J �t|tj�s4J �|jdks;J �~~t� � t58|� t ||�}||�}t |�dksVJ �d} | |d v s`J �t|tj�shJ �|jdksoJ �d S )59z8Check that mmap_mode is respected even at the first callrI r r� c S s | d S �Nr r �r� r r r �twice� � z0test_memory_numpy_check_mmap_mode.<locals>.twicer% r � Exception while loading resultsN)
r rQ r& r Zones�60isinstanceZmemmapr_ �gcZcollectr9 rB r( )61rW � monkeypatchr+ r� r� r� �c�recorded_warnings�d�
exception_msgr r r �!test_memory_numpy_check_mmap_mode� s* 626364r� c sz t | jdd�}G dd� dt�� |jd65� fdd� �}|� td�D ]}t� �� |d� W d � n1 s5w Y q d S ) z+Smoketest the exception handling of Memory.r r" c @ � e Zd ZdS )z*test_memory_exception.<locals>.MyExceptionN��__name__rV � __qualname__r r r r � MyException� � r� c s | r� �d S r: r )�exc�r� r r �h� s �z test_memory_exception.<locals>.hr% r N)r )r rQ � Exceptionr& r' r )rW r+ r� r- r r� r �test_memory_exception� s 6667 ���r� c s� t | jdd�}t� � |jdgd�d � fdd� �}|jdgks J �|ddd� t� �dks.J �|ddd� t� �dks<J �|dd d� t� �dksJJ �d68S ) z!Test the ignore feature of memoryr r" r ��ignorer c � � � d� d S rD r; r rG r r �z� � z test_memory_ignore.<locals>.z�r r N�r �r rQ rP r& r� r( )rW r+ r� r rG r �test_memory_ignore� s r� c s� t | jdd�}t� � dd� }|jdgd�|d
� fdd � ��}|jdgks&J �|ddd69� t� �dks4J �|ddd70� t� �dksBJ �|dd d71� t� �dksPJ �d S )z9Test the ignore feature of memory on a decorated functionr r" c s t �� �� fdd��}|S )Nc s � | i |��S r: r )�args�kwargs�r! r r �wrapped� r� z?test_memory_ignore_decorated.<locals>.decorate.<locals>.wrapped)r� �wraps)r! r� r r� r �decorate� s z.test_memory_ignore_decorated.<locals>.decorater r� r c r� rD r; r rG r r r� � s z'test_memory_ignore_decorated.<locals>.zr� r Nr� r� )rW r+ r� r� r rG r � test_memory_ignore_decorated� s r� c C sT t | jdd�}|jdd� �}|d�dksJ �|dd�dks J �|dd�dks(J �d S )72z_Non-regression test against 0.12.0 changes.73 74 https://github.com/joblib/joblib/pull/75175 r r" c S r� rD r r� r r r �plus_one r� z,test_memory_args_as_kwargs.<locals>.plus_oner r r� r% N)r rQ r& )rW r+ r� r r r �test_memory_args_as_kwargs s 76r� zignore, verbose, mmap_moder �d rJ c C sT t | jdd�}|j|||d�dd� �}|j|ksJ �|j|ks!J �|j|ks(J �dS )z7Check cache may be called with kwargs before decoratingr r" )r� r$ rK c S � d S r: r r| r r r r� ' � z"test_partial_decoration.<locals>.zN)r rQ r& r� Z_verboserK )rW r� r$ rK r+ r� r r r �test_partial_decoration" s 77r� c C s* t | jdd�}t�d�}|�d� | jdg|�R � j}|�t�}tt�}t j78�|j j |�}||ks3J �t j79�
|�s;J �|j t j80�|j j �ksHJ �t�� |�� rRJ �t j81�
t j82�|d��s_J �|�� seJ �|�d�}t j83�|j j |j|�}|d�}t j84�
|�s�J �t �t j85�|d�� ||d�ks�J �d S ) Nr r" �.r! r` z func_code.pyr r/ )r rQ r� �splitr< r6 r& r! r r4 r5 r2 r# �exists�dirnamer rT �_check_previous_func_code� _get_args_id�func_id�remove) rW r+ r5 rX r� r# �args_id�86output_dirr� r r r �
test_func_dir0 s( 878889 90r� c C s� t | jdd�}|�t�}|d�}t�t�|��}|�d�}tj �91|j j |j
|�}tj �|�s0J �||j �|j
|g�ks=J �t�t�|��}|j j |j j ksOJ �t d dd�}t�t�|�� |�t�}t�t�|��}|d� d S )Nr r" r )r rQ r& r! �pickle�loads�dumpsr� r4 r5 r6 r2 r# r� r� Z load_item) rW r+ rX �outputr� r� r� Zmemory2�gpr r r �test_persistenceP s 9293 94 r� �consider_cache_validc s� t t| j� fdd�d�t| jdd�jt� fdd�d�fD ]0}|�d�}t|t�s*J �|r.J �|d�dks6J �|�d�}t|t�sBJ �|� ksHJ �|�� q t t�}|�d�rXJ �d S ) Nc � � S r: r �r- �r� r r r k r� z*test_check_call_in_cache.<locals>.<lambda>��cache_validation_callbackr r" c r� r: r r� r� r r r n r� r � )95r r! rQ r r& Zcheck_call_in_cacher� �boolrT r )rW r� r) �resultr r� r �test_check_call_in_cacheg s" � ��9697 98r� c C s� t tt| j�tt�t| jdd��t�td d��t�fttttf�D ]@\}}|d�dks-J �|� d�}t99||�s9J �|� � dksAJ �|� � t
t�� |� � W d � n1 sXw Y |� � q!d S )Nr r" �r# r r� )�zipr r! rQ r r r& r100 r
�call_and_shelver� �getrT r �KeyError)rW r) �Resultr� r r r �test_call_and_shelve~ s$ 101�102 � 103104105 �106�r� c C s | � d�}|�d� t�|j�j}t�d� |�� dks J �|t�|j�jkr,t �107d� t | jdd�}|� t
�}|�d�}tj� |jj|j|d�}|d�dksQJ �t�|�j}t�d � |�d�}t|t�shJ �t�|�j|ksrJ �t�d � |�� dksJ �t�|�j|ks�J �d108S ) z6Check call_and_shelve only load stored data if needed.Z test_access� �?z>filesystem does not support fine-grained access time attributer r" r r/ r� r N)r6 r8 r4 �statrQ �st_atimers rt �read�pytest�skipr r& r! r� r5 r2 r# r� r� r� r109 r� ) rW Ztest_access_time_fileZtest_access_timer+ r) r� � result_pathZfirst_access_timer� r r r �.test_call_and_shelve_lazily_load_stored_result� s, 110111112113114115� 116117118r� c C s� t t| j�tt�fD ]Q}| �d�j}|�d�}t|d��}t�||� W d � n1 s-w Y t|d��
}t� |�}W d � n1 sGw Y |�119� |�120� ksVJ �t � |� q121d S )Nzpickling_test.datr �wb�rb)
r r! rQ r r6 r� r7 r� �dump�loadr� r4 r� )rW r) r^ r� �fp�result2r r r �test_memorized_pickling� s 122 � � �r� c C s123 t t| j�}|�d�}t t| j�}|�d�}|�� |�� ks J �t|�t|�ks*J �tt�}t|� t|�d�� t t| jdt�� d�}|�d�}|�� t t| jdd�}|�d�}|�� t t| jdt�� d�}|�d�}|�� t t| jdd�}|�d�}|�� d S )Nr � )r$ � timestamp)r$ r� )r r! rQ r� r� �reprr rs )rW r) r� Zfunc2r� r r r �test_memorized_repr� s* 124 125126127128129 r c C s� |� d�j}tj�|d�}d}t|d��
}|�|� W d � n1 s$w Y |�|� dd l}t |jdd�}|�130|j �} | d� | d� | d� t|d��}|�d | � W d � n1 sbw Y | d� | d� t �
|� t� |� d131}t|d��
}|�|� W d � n1 s�w Y | d� | d� tj�d � tj�d � dd l}|�132|j �} | d� | d� | �� \}133} |134d
ks�J �d S )NZ135tmp_importztmp_joblib_.pyz$def f(x):136 print(x)137 return x138r0 r r" r r z139 140z-def f(x):141 print("x=%s" % x)142 return x143z144Reloading145� tmp_joblib_z11462147Reloading148x=1149)�mkdirrQ r4 r5 r6 r7 r8 Zsyspath_prependr r r& r! rR rS �sys�stdout�modules�pop�150readouterr) ZcapsysrW r� �dir_namer^ �contentZ module_file�tmpr+ r! rZ �errr r r � test_memory_file_modification� sD �151 �152153 � r c C r� r: r �r� r� r r r �_function_to_cache r� r c C � | | S r: r r r r r �_sum# � r c C s | | S r: r r r r r �_product' r r c C s� t jt_t| jdd�}|�t�}|dd�dksJ �|dd�dks"J �tt�� tjt_|dd�dks4J �|dd�dks=J �W d � d S 1 sHw Y d S )Nr r" r r r% ) r rp r r rQ r& r r r )rW r+ r! r r r �*test_memory_in_memory_function_code_change+ s 154155"�r c C s t d d�} | �� d S )Nr� )r rT )r+ r r r �$test_clear_memory_with_none_location; s 156 r �kw1�kw2�r r c C s | |||fS r: r )r� r� r r r r r �func_with_kwonly_args@ s r r� r� �returnc C r r: r r r r r �func_with_signatureD r r c C s t | jdd�}|�t�}|dddd�dksJ �tt��}|ddddd � W d � n1 s.w Y |�d157� |ddddd � tt��}|ddddd � W d � n1 sWw Y |�d158� |jtd gd
�}|ddddd �dksuJ �|ddddd �dks�J �d S )Nr r" r r r% )r )r r r% r rk )r z?Keyword-only parameter 'kw1' was passed as positional parameterr r r� )r r r% rk Zignored)r rQ r& r r �159ValueError�match)rW r+ � func_cached�excinfor r r �!test_memory_func_with_kwonly_argsH s 160161 �162163 �164 r" c C s. t | jdd�}|�t�}|dd�dksJ �d S )Nr r" r g @g @)r rQ r& r )rW r+ r r r r � test_memory_func_with_signatureb s 165r# c st t | jdd�}|�� dd� ��tt|��}|D ]}�|� qt��� �fdd�|D �}� �fdd�|D �}||�fS )Nr r" c S � dS )N�� aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaar rE r r r �get_1000_bytesl r� z(_setup_toy_cache.<locals>.get_1000_bytesc s g | ]}� � |��qS r )r� )re rF )r& r r �166<listcomp>u � z$_setup_toy_cache.<locals>.<listcomp>c s g | ] }t j��jj� |��qS r )r4 r5 r6 r2 r# )re r� �r� r& r r r' w s ��)r rQ r& rP r' r )rW �167num_inputsr+ ZinputsrF Z
hash_dirnamesZ
full_hashdirsr r) r �_setup_toy_cachei s 168 169 �170r+ c s� t | �\}}}|j�� }dd� |D �}t|�t|�ks J �dd� � � fdd�|D �}dd� |D �}||ks7J �dd� |D �}dd� |D �} d d� |D �}171|172| ksRJ �d S )173Nc S � g | ]}|j �qS r )r5 �re �cir r r r' � � z#test__get_items.<locals>.<listcomp>c s* � fdd�t �� �D �}tdd� |D ��S )Nc s g | ] }t j�� |��qS r �r4 r5 r6 �re �fn�� directoryr r r' � � z;test__get_items.<locals>.get_files_size.<locals>.<listcomp>c s s � | ] }t j�|�V qd S r: )r4 r5 �getsize)re r� r r r rg � rh z:test__get_items.<locals>.get_files_size.<locals>.<genexpr>)r4 rr �sum)r4 Z174full_pathsr r3 r �get_files_size� s z'test__get_items.<locals>.get_files_sizec s g | ]}� |��qS r r �re Zhash_dir�r8 r r r' � s c S r, r )�sizer- r r r r' � r/ c S s g | ] }t j�|d ��qS )r/ r0 r9 r r r r' � r5 c S s g | ] }t j �tj�|���qS r )�datetime�
fromtimestampr4 r5 �getatimer1 r r r r' � s �c S r, r �Z last_accessr- r r r r' � r/ )r+ r2 r3 �set) rW r+ Zexpected_hash_dirsr- �itemsZ hash_dirsZexpected_hash_cache_sizesZhash_cache_sizesZoutput_filenamesZexpected_last_accessesZ
last_accessesr r: r �test__get_items~ s 175 �rB c
C s$ t | dd�\}}}|j�d�}|g ksJ �t | �\}}}|j�� }|j�d�}t|�}t�t|�t|��s7J �t|�|d ksAJ �|j�d�}t|�t|�ksQJ �|j�d�}|g ks]J �d} |j�| �}176t|177�soJ t|���|j�d �} t|��| �} t d178d � | D ��t179d180d � | D ��ks�J �d S ) Nr )r* Z1KZ2Kr i �1M�� ip c s s � | ]}|j V qd S r: r? r- r r r rg � s � z,test__get_items_to_delete.<locals>.<genexpr>) r+ r2 Z_get_items_to_deleter3 r( r@ �issubset�sorted�181difference�max�min)
rW r+ r- Zitems_to_deleteZexpected_hash_cachedirsrA Z nb_hashesZitems_to_delete_2048bZitems_to_delete_empty�bytes_limit_too_smallZitems_to_delete_500bZitems_to_delete_6000bZsurviving_itemsr r r �test__get_items_to_delete� s. 182 � �rK c C s� t | �\}}}|j�� }|�� |j�� }t|�t|�ks J �|jdd� |j�� }t|�t|�ks4J �|jdd� |j�� }t�t|�t|��sKJ �t|�dksSJ �d}|j|d� |j�� }|g ksfJ �d S )NrC )Z bytes_limitZ3Kr rD �r+ r2 r3 � reduce_sizerF r@ rE r( )rW r+ r- �ref_cache_items� cache_itemsrJ r r r �#test_memory_reduce_size_bytes_limit� s 183184 185 186 187rP c C s� t | �\}}}|j�� }|�� |j�� }t|�t|�ks J �|jdd� |j�� }t|�t|�ks4J �|jdd� |j�� }t�t|�t|��sKJ �t|�dksSJ �|jdd� |j�� }|g ksdJ �d S )NrJ )Z items_limitr r rL )rW r+ r- rN rO r r r �#test_memory_reduce_size_items_limit� s 188189 190 191 192rQ c C sZ dd l }dd l}t| �\}}}|j�� }|�� |j�� }t|�t|�ks'J �|j|jdd�d� |j�� }t|�t|�ks@J �|�d� |d� |d� |j|jdd�d� |j�� }t �193t |�t |��rhJ �t |�dkspJ �t j
td d194�� |j|jdd�d� W d � n1 s�w Y |�d � |j|jdd�d� |j�� }|g ks�J �d S ) Nr r )�days)Z age_limit������������secondsr zhas to be a positive�r g����MbP?)r< rs r+ r2 r3 rM rF � timedeltart r@ rE r( r� r r )rW r< rs r+ r- Z put_cacherN rO r r r �!test_memory_reduce_size_age_limit� s0 195196197198199 �200201rY c C s@ t | �\}}}|�� t�|jj�g ksJ �|jdd�r J �d S )Nrk )�202stacklevel)r+ rT r4 rr r2 r# r� )rW r+ r- rX r r r �test_memory_clear s r[ c C s dgd } | S )Nr% i� r )� complex_objr r r � fast_func_with_complex_output# rH r] c C s" dd� t td��D �}| r|S dS )Nc S s i | ]}t |�|�qS r )r� rd r r r �203<dictcomp>) r( z=fast_func_with_conditional_complex_output.<locals>.<dictcomp>g j�@z
simple output)r' �int)Zcomplex_outputr\ r r r �)fast_func_with_conditional_complex_output( s r` c �b t | jd�}|�t�� tdd�� fdd�td�D �� |�� \}}d}||vs)J �||vs/J �d S )Nr� r ra c 3 s � | ]}t � �� V qd S r: rc rd �r r r rg 5 s � zMtest_cached_function_race_condition_when_persisting_output.<locals>.<genexpr>r% r� )r rQ r& r] r r' r204 �rW Zcapfdr+ r �stderrr� r rb r �:test_cached_function_race_condition_when_persisting_output- s 205 re c ra )Nr� r ra c 3 s, � | ]}t � �|d dkrdnd�V qdS )r r TFNrc rd rb r r rg K s � 206�zOtest_cached_function_race_condition_when_persisting_output_2.<locals>.<genexpr>r% r� )r rQ r& r` r r' r207 rc r rb r �<test_cached_function_race_condition_when_persisting_output_2A s 208� rf c 209 C s� t | jd�}dd� }|�|�}d}||�\}}||ks J �t|� t||�}||�\} }210t|�dks4J �d} | |d v s>J �| |ksDJ �|211|ksJJ �t|� |�|�} z| �� td�� t yy }
zd }|t212|
j �v snJ �W Y d }
~
d S d }
~
ww )213Nr� c S s t �d� | t � � fS )Ng{�G�z�?�rs rt rE r r r r) \ s 214 zItest_memory_recomputes_after_an_error_while_loading_results.<locals>.funcrF r r� r z<It normally not possible to load a corrupted MemorizedResultz is corrupted) r rQ r& r9 rB r( r� r� �AssertionErrorr� r� r� )rW r� r+ r) Z cached_funcZ input_argrF r r� Zrecomputed_argZrecomputed_timestampr� � reference�er� r r r �;test_memory_recomputes_after_an_error_while_loading_resultsY s4 215 216 217� ��rk c @ s e Zd ZdZdS )�IncompleteStoreBackendzAThis backend cannot be instantiated and should raise a TypeError.N)r� rV r� �__doc__r r r r rl � s rl c @ sP e Zd ZdZdd� Zdd� Zdd� Zdd � Zd218d � Zd d
� Z dd� Z219dd� Z dS )�DummyStoreBackendz(A dummy store backend that does nothing.c O r$ )zOpen an item on store.Nr )�selfr� r� r r r �220_open_item� r� z DummyStoreBackend._open_itemc C r$ )z!Check if an item location exists.Nr �ro r# r r r � _item_exists� r� z DummyStoreBackend._item_existsc C r$ )z&Move an item from src to dst in store.Nr )ro �src�dstr r r �221_move_item� r� z DummyStoreBackend._move_itemc C r$ )zCreate location on store.Nr rq r r r �create_location� r� z!DummyStoreBackend.create_locationc C r$ )z&Check if an object exists in the storeFr �ro �objr r r r� � r� zDummyStoreBackend.existsc C r$ )zClear object on storeNr rw r r r �clear_location� r� z DummyStoreBackend.clear_locationc C s g S )z3Returns the whole list of items available in cache.r )ro r r r r3 � r� zDummyStoreBackend.get_itemsc O r$ )zConfigure the storeNr )ro r# r� r� r r r � configure� r� zDummyStoreBackend.configureN) r� rV r� rm rp rr ru rv r� ry r3 rz r r r r rn � s rn �invalid_prefixc C s@ t t��
}t| d � W d � n1 sw Y |�d� d S )Nz&Store backend name should be a string*�r r r r )r{ r! r r r �(test_register_invalid_store_backends_key� � 222 �r} c C s@ t t��
} tdd � W d � n1 sw Y | �d� d S )N�fsz.Store backend should inherit StoreBackendBase*r| �r! r r r �+test_register_invalid_store_backends_object� r~ r� c C sB t t��} tddd� W d � n1 sw Y | �d� d S )Nz /tmp/joblib�unknown)r# �backendzUnknown location*)r � TypeErrorr r r� r r r �!test_memory_default_store_backend� s 223 �r� c C sf G dd� d�} | � }t t��}td|d� W d � n1 s w Y d}|t|d j�v s1J �d S )Nc @ r� )zHtest_warning_on_unknown_location_type.<locals>.NonSupportedLocationClassNr� r r r r �NonSupportedLocationClass� r� r� �localr� zbInstantiating a backend using a NonSupportedLocationClass as a location is not supported by joblibr )r � UserWarningr r� r� )r� Zunsupported_locationr� Zexpected_mesager r r �%test_warning_on_unknown_location_type� s 224 ��r� c C sb d} t | t� | tft�� v sJ �tt��
}t| d� W d � n1 s%w Y |�d� d S )NZisbZ
fake_locationznCan't instantiate abstract class IncompleteStoreBackend (without an implementation for|with) abstract methods*)r rl r rA r r� r r )� backend_namer! r r r �)test_instanciate_incomplete_store_backend� s 225226 ��r� c C s> d} t | t� | tft�� v sJ �t| d�}t|t�s J �d S )NZdsbZdummy_location)r rn r rA r r� )r� � backend_objr r r �test_dummy_store_backend� s227 228229r� c 230 C sF t d�} td| �}z|jdksJ �W tjddd� d S tjddd� w )NZ some_folderr� TrM )r r r# rR rS )r5 r� r r r �0test_instanciate_store_backend_with_pathlib_path� s231 232"r� c C sj d}t � }|jd u s J �t|� t|�|jd d�ks J �|�| j� t|�|j| jd�ks/J �t|� d S )Nz-FileSystemStoreBackend(location="{location}")r� )r r# r r� rl rz rQ )rW Z repr_patternr� r r r �"test_filesystem_store_backend_repr� s r� c C s� dd� }t | jdd�}|�|�}d}t|�|j||jjd�ks!J �|�dd�}d}t|�|j|jj|j|j d �ks;J �t|�d233j|jjd �ksIJ �d S ) Nc S r r: r r r r r �my_func r z)test_memory_objects_repr.<locals>.my_funcr r" z/MemorizedFunc(func={func}, location={location}))r) r# �* zJMemorizedResult(location="{location}", func="{func}", args_id="{args_id}"))r# r) r� zMemory(location={location})r� )234r rQ r& r� rl r2 r# r� r� r� )rW r� r+ Zmemorized_funcZmemorized_func_repr�memorized_resultZmemorized_result_reprr r r �test_memory_objects_repr s$ 235236 � �237 �238�r� c C s� t | jd�}|jdd� �}|�d�}t�|�}t�|�}|jj|jjks&J �|j |j ks.J �|j239|j240ks6J �t |�t |�ks@J �d S )Nr� c S � | d S r� r r| r r r rX 6 r� z'test_memorized_result_pickle.<locals>.grk ) r rQ r& r� r� r� r� r2 r# r) r� r� )rW r+ rX r� Zmemorized_result_pickleZmemorized_result_loadsr r r � test_memorized_result_pickle/ s 241242243244�r� c C sh |d u rg }t | �}t |�}t|�� �t|�� �ks J �|�� D ]}||v r'q || || ks1J �q d S r: )�varsr@ �keys)�left�right�
ignored_attrsZ left_varsZ245right_vars�attrr r r �compareG s �r� �
memory_kwargsr% r )rL r$ r� Z parameterZunused)rK r$ Zbackend_optionsc C s t dd| ji|��}t�t�|��}t|j|j� t||tg d��d� t|�t|�ks.J �|� t246�}t�t�|��}t|j|j� t||tg d��d� t|�t|�ksWJ �|� d�}t�t�|��}t|j|j� t||tg d��d� t|�t|�ks�J �d S )Nr# )r2 r Z
_func_code_id)r� r r ) r rQ r� r� r� r� r2 r@ r r& r! r� )rW r� r+ Zmemory_reloadedr Zfunc_cached_reloadedr� Zmemorized_result_reloadedr r r � test_memory_pickle_dump_loadT s6 247�248249�250251�r� c C s� |� tj� d}t| jdd�}|jdd� �}||�}d|jv s!J �|�� t| jdd�}|jdd� �}||�}d|jvs>J �|�� d S ) Nr% � r" c S r� r� r r| r r r r! � r� ztest_info_log.<locals>.fZQueryingr c S r� r� r r| r r r r! � r� )Z set_level�logging�INFOr rQ r& �textrT )rW Zcaplogr r+ r! r- r r r �
test_info_log� s 252253 r� c @ sL e Zd ZdZddd�Zdd� Zej�ddd g�d254d � �Z d d
� Z255dd� Z dS )�TestCacheValidationCallbackz.Tests on parameter `cache_validation_callback`Nc C s" d|d< |d ur
t �|� |d S )NT�runr rg )ro r r� �delayr r r �foo� s 256z TestCacheValidationCallback.fooc C sD d}t jt|d�� |jdd� W d � dS 1 sw Y dS )z2Test invalid values for `cache_validation_callbackz9cache_validation_callback needs to be callable. Got True.rW Tr� N)r� r r r& )ro r+ r r r r �&test_invalid_cache_validation_callback� s "�zBTestCacheValidationCallback.test_invalid_cache_validation_callbackr� TFc sr |j | j� fdd�dgd�}ddiddi}}|d|�dks J �|d|�dks)J �|d s/J �|d � ks7J �d S )257zTest expiry of old resultsc r� r: r r� r� r r r � r� zUTestCacheValidationCallback.test_constant_cache_validation_callback.<locals>.<lambda>r� �r� r� r� Fr rk N�r& r� )ro r+ r� r! �d1�d2r r� r �'test_constant_cache_validation_callback� s 258� zCTestCacheValidationCallback.test_constant_cache_validation_callbackc C s� dd� }|j | j|dgd�}ddiddi}}|d|dd �d259ks"J �|d|dd �d260ks-J �|d s3J �|d s9J �ddiddi}}|d|d d �d261ksMJ �|d|d d �d262ksXJ �|d s^J �|d rdJ �d S )
z*Test cache validity based on run duration.c S s | d }|dkr263dS d S )N�durationg�������?Tr )�metadatar� r r r r� � s �z^TestCacheValidationCallback.test_memory_only_cache_long_run.<locals>.cache_validation_callbackr� r� r� Fr r )r� rk rj Nr� )ro r+ r� r! r� r� r r r � test_memory_only_cache_long_run� s 264� z;TestCacheValidationCallback.test_memory_only_cache_long_runc C s� |j | jtdd�dgd�}ddiddiddi}}}|d|�dks$J �|d|�dks-J �t�d � |d|�dks;J �|d sAJ �|d rGJ �|d sMJ �d265S ) z!Test expiry of old cached resultsg333333�?rU r� r� r� Fr rk r� N)r& r� r rs rt )ro r+ r! r� r� Zd3r r r �test_memory_expires_after� s � 266 z5TestCacheValidationCallback.test_memory_expires_afterr: ) r� rV r� rm r� r� r� �markr r� r� r� r r r r r� � s 267268 r� c @ s, e Zd ZdZedd� �Zdd� Zdd� ZdS ) �TestMemorizedFunc�8Tests for the MemorizedFunc and NotMemorizedFunc classesc C s |� | d�d || < || S )Nr r )r� )r �counterr r r r! � s zTestMemorizedFunc.fc C sp |j | jdgd�}i }|d|�dksJ �|d|�dks J �|�d|�\}}|dks-J d��t|t�s6J d��dS )�Test calling the functionr� r� r r � f has not been called properly�0Metadata are not returned by MemorizedFunc.call.N)r& r! �callr� �dict�ro r+ r! r� r �metar r r �test_call_method_memorized� s �z,TestMemorizedFunc.test_call_method_memorizedc C sh t | j�}i }|d|�dksJ �|d|�dksJ �|�d|�\}}|dks)J d��t|t�s2J d��dS )r� r r r% r� r� N)r r! r� r� r� r� r r r � test_call_method_not_memorized� s 269 �z0TestMemorizedFunc.test_call_method_not_memorizedN)r� rV r� rm � staticmethodr! r� r� r r r r r� � s 270 r� c @ s e Zd ZdZdd� Zdd� ZdS )�TestAutoGitignorer� c C sP t |�d��}t|�}|�t�}|d� |d }|�� s J �|�� dks&J �dS )zsTest that using the memory object automatically creates a `.gitignore` file271 within the new cache directory.�test_cache_dirr �272.gitignorez%# Created by joblib automatically.273*274N)r r r r& �idr� � read_text)ro rW r# �memZcostly_operation�gitignore_filer r r � test_memory_creates_gitignore s 275 z/TestAutoGitignore.test_memory_creates_gitignorec C sZ t |�d��}|d }d}|�|� t|�}|�t�d� |�� s#J �|�� |ks+J �dS )zvTest that using the memory object does not overwrite an existing276 `.gitignore` file within the cache directory.r� r� z# Existing .gitignore file!r N)r r �277write_textr r& r� r� r� )ro rW r# r� Zexisting_contentr� r r r �1test_memory_does_not_overwrite_existing_gitignore s 278 zCTestAutoGitignore.test_memory_does_not_overwrite_existing_gitignoreN)r� rV r� rm r� r� r r r r r� s r� r� )rJ )Tr: )wrm r< r� r� r� r4 Zos.pathr� rR r rn rs �pathlibr r� Zjoblib._store_backendsr r Zjoblib.hashingr Z
joblib.memoryr r r r r279 r r r
r r r r Zjoblib.parallelr r Zjoblib.test.commonr r r Zjoblib.testingr r r r! r. r9 rB r[ ru rw rz r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r r r r r r r r r_ �floatr r" r# r+ rB rK rP rQ rY r[ r] r` re rf rk rl rn r� rP r} r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r r r r �<module> s� 8280 .281i 282283 284'285
286$ 9287( %288 289290291-$292 293 294
��295(K'