CoolFace
Apppublic

beatricechen107/Pathology_TGH

sourceHugging Faceupdated 1y agoView on Hugging Face
0likes
commit_scheduler.cpython-311.pyc199 linesDownload Raw Back to __pycache__
1�

2:��hV=��B�ddlZddlZddlZddlZddlmZddlmZddlm	Z	m3Z4mZddlm
Z
ddlmZmZddlmZmZmZmZmZddlmZmZmZmZdd	lmZeje��Z ed5���Gd�d
����Z!Gd�d��Z"Gd�de��Z#dS)�N)�Future)�	dataclass)�SEEK_END�SEEK_SET�BytesIO)�Path)�Lock�Thread)�Callable�Dict�List�Optional�Union)�DEFAULT_IGNORE_PATTERNS�6CommitInfo�CommitOperationAdd�HfApi)�filter_repo_objectsT)�frozenc�<�eZdZUdZeed<eed<eed<eed<dS)�
_FileToUploadzWTemporary dataclass to store info about files to upload. Not meant to be used directly.�7local_path�path_in_repo�8size_limit�
last_modifiedN)	�__name__�9__module__�__qualname__�__doc__r�__annotations__�str�int�float���W/home/jma/anaconda3/envs/mSTAR/lib/python3.11/site-packages/trackio/commit_scheduler.pyrrsE�������a�a����������O�O�O������r%rc��eZdZdZdddddddddddd�dedeeefdeeefd	e	ed10e	ede	ede	e11d
e	ede	eeeefde	eeeefde12de	dde	egdfddfd�Z
dd�Zdd�Zdd�Zdd�Zdefd�Zde	efd�Zde	efd�ZdS)�CommitSchedulera�
13    Scheduler to upload a local folder to the Hub at regular intervals (e.g. push to hub every 5 minutes).14 15    The recommended way to use the scheduler is to use it as a context manager. This ensures that the scheduler is16    properly stopped and the last commit is triggered when the script ends. The scheduler can also be stopped manually17    with the `stop` method. Checkout the [upload guide](https://huggingface.co/docs/huggingface_hub/guides/upload#scheduled-uploads)18    to learn more about how to use it.19 20    Args:21        repo_id (`str`):22            The id of the repo to commit to.23        folder_path (`str` or `Path`):24            Path to the local folder to upload regularly.25        every (`int` or `float`, *optional*):26            The number of minutes between each commit. Defaults to 5 minutes.27        path_in_repo (`str`, *optional*):28            Relative path of the directory in the repo, for example: `"checkpoints/"`. Defaults to the root folder29            of the repository.30        repo_type (`str`, *optional*):31            The type of the repo to commit to. Defaults to `model`.32        revision (`str`, *optional*):33            The revision of the repo to commit to. Defaults to `main`.34        private (`bool`, *optional*):35            Whether to make the repo private. If `None` (default), the repo will be public unless the organization's default is private. This value is ignored if the repo already exists.36        token (`str`, *optional*):37            The token to use to commit to the repo. Defaults to the token saved on the machine.38        allow_patterns (`List[str]` or `str`, *optional*):39            If provided, only files matching at least one pattern are uploaded.40        ignore_patterns (`List[str]` or `str`, *optional*):41            If provided, files matching any of the patterns are not uploaded.42        squash_history (`bool`, *optional*):43            Whether to squash the history of the repo after each commit. Defaults to `False`. Squashing commits is44            useful to avoid degraded performances on the repo when it grows too large.45        hf_api (`HfApi`, *optional*):46            The [`HfApi`] client to use to commit to the Hub. Can be set with custom settings (user agent, token,...).47        on_before_commit (`Callable[[], None]`, *optional*):48            If specified, a function that will be called before the CommitScheduler lists files to create a commit.49 50    Example:51    ```py52    >>> from pathlib import Path53    >>> from huggingface_hub import CommitScheduler54 55    # Scheduler uploads every 10 minutes56    >>> csv_path = Path("watched_folder/data.csv")57    >>> CommitScheduler(repo_id="test_scheduler", repo_type="dataset", folder_path=csv_path.parent, every=10)58 59    >>> with csv_path.open("a") as f:60    ...     f.write("first line")61 62    # Some time later (...)63    >>> with csv_path.open("a") as f:64    ...     f.write("second line")65    ```66 67    Example using a context manager:68    ```py69    >>> from pathlib import Path70    >>> from huggingface_hub import CommitScheduler71 72    >>> with CommitScheduler(repo_id="test_scheduler", repo_type="dataset", folder_path="watched_folder", every=10) as scheduler:73    ...     csv_path = Path("watched_folder/data.csv")74    ...     with csv_path.open("a") as f:75    ...         f.write("first line")76    ...     (...)77    ...     with csv_path.open("a") as f:78    ...         f.write("second line")79 80    # Scheduler is now stopped and last commit have been triggered81    ```82    �NF)�everyr�	repo_type�revision�private�token�allow_patterns�ignore_patterns�squash_history�hf_api�on_before_commit�repo_id�folder_pathr*rr+r,r-r.r/r0r1r2rr3�returnc
	���|pt|���|_|
|_t|��������|_|pd|_|	|_|83�g}84nt|85t��r|86g}87|88tz|_|j�
��rtd|j�d����|j�dd���|j�|||d���}|j|_||_||_||_i|_|dkstd	|�d����t-��|_||_||_t4�d89|j�d|j�d|j�d
���t9|jd���|_|j���tAj!|j"��d|_#dS)N)r.�z0'folder_path' must be a directory, not a file: '�'.T)�parents�exist_ok)r4r-r+r;rz)'every' must be a positive integer, not 'zScheduled job to push 'z' to 'z' every z	 minutes.)�target�daemonF)$r�apir3r�90expanduser�resolver5rr/�91isinstancer!rr0�is_file�92ValueError�mkdir�create_repor4r+r,r.�
last_uploadedr	�lockr*r1�logger�infor93�_run_scheduler�_scheduler_thread�start�atexit�register�_push_to_hub�_CommitScheduler__stopped)�selfr4r5r*rr+r,r-r.r/r0r1r2r3�repo_urls               r&�__init__zCommitScheduler.__init__ls
��"�/�U��/�/�/��� 0��� ��,�,�7�7�9�9�A�A�C�C���(�.�B���,����"� �O�O�
���
-�
-�	0�.�/�O�.�1H�H�����#�#�%�%�	��W�4�CS�W�W�W���
�	
����t�d��;�;�;��8�'�'��W�	�D�(�94�95�� �'���"��� ��
���96�97
�	
��98�q�y�y��R��R�R�R�S�S�S��F�F��	���99�,������i�d�&6�i�i�d�l�i�i�TX�T^�i�i�i�	100�	101�	102�"(�t�/B�4�!P�!P�!P�����$�$�&�&�&����)�*�*�*�����r%c��d|_dS)ziStop the scheduler.103 104        A stopped scheduler cannot be restarted. Mostly for tests purposes.105        TN)rP�rQs r&�stopzCommitScheduler.stop�s��106����r%c��|S�Nr$rUs r&�	__enter__zCommitScheduler.__enter__�s���r%c�z�|������|���dSrX)�trigger�resultrV)rQ�exc_type�	exc_value�	tracebacks    r&�__exit__zCommitScheduler.__exit__�s.�������������	�	�����r%c��	|���|_tj|jdz��|jrdS�?)z7Dumb thread waiting between each scheduled push to Hub.T�<N)r[�last_future�time�sleepr*rPrUs r&rJzCommitScheduler._run_scheduler�sA��	�#�|�|�~�~�D���J�t�z�B��'�'�'��~�
���		r%c�@�|j�|j��S)z�Trigger a `push_to_hub` and return a future.107 108        This method is automatically called every `every` minutes. You can also call it manually to trigger a commit109        immediately, without waiting for the next scheduled commit.110        )r>�
run_as_futurerOrUs r&r[zCommitScheduler.trigger�s���x�%�%�d�&7�8�8�8r%c�p�|jrdSt�d��	|���}|jrFt�d��|j�|j|j|j	���|S#t$r#}t�d|�����d}~wwxYw)Nz((Background) scheduled commit triggered.z$(Background) squashing repo history.)r4r+�branchzError while pushing to Hub: )rPrHrI�push_to_hubr1r>�super_squash_historyr4r+r,�	Exception�error)rQ�value�es   r&rOzCommitScheduler._push_to_hub�s����>�	��4����>�?�?�?�	��$�$�&�&�E��"�
����B�C�C�C���-�-� �L�D�N�4�=�.�����L���	�	�	��L�L�2�q�2�2�
�
�
�
�����		���s�A"B�111B5�B0�0B5c112�4���j5�j�����t�d���fd�t	�j�d����D��}�jr�j�d���d�nd}g}t|�113���j�j���D]�}||}|�
��}�j�|����j||jkr3|�t%|||z|j|j�������	ddd��n#1swxYwYt)|��d	krt�d114��dSt�d��d�|D��}t�d
���j��j�j|d�j���}|D]}	|	j�j|	j<�|S)a�115        Push folder to the Hub and return the commit info.116 117        <Tip warning={true}>118 119        This method is not meant to be called directly. It is run in the background by the scheduler, respecting a120        queue mechanism to avoid concurrent commits. Making a direct call to the method might lead to concurrency121        issues.122 123        </Tip>124 125        The default behavior of `push_to_hub` is to assume an append-only folder. It lists all files in the folder and126        uploads only changed files. If no changes are found, the method returns without committing anything. If you want127        to change this behavior, you can inherit from [`CommitScheduler`] and override this method. This can be useful128        for example to compress data together in a single file before committing. For more details and examples, check129        out our [integration guide](https://huggingface.co/docs/huggingface_hub/main/en/guides/upload#scheduled-uploads).130        Nz-Listing files to upload for scheduled commit.c���i|]C}|����|��j�����|��DSr$)rB�relative_tor5�as_posix)�.0�pathrQs  �r&�131<dictcomp>z/CommitScheduler.push_to_hub.<locals>.<dictcomp>�sX���"�"�"���<�<�>�>�"�� � ��!1�2�2�;�;�=�=�t�"�"�"r%z**/*�/r8)r/r0)rrrrrz4Dropping schedule commit: no changed file to upload.z9Removing unchanged files since previous scheduled commit.c�D�g|]}t|j|j�����S))�path_or_fileobjr)rrr)rt�file_to_uploads  r&�132<listcomp>z/CommitScheduler.push_to_hub.<locals>.<listcomp>"sG��133�134�135��

�!/� 9�+�8�	
�
�
�136�137�138r%z%Uploading files for scheduled commit.zScheduled Commit)r4r+�139operations�commit_messager,)rGr3rH�debug�sortedr5�globr�stripr�keysr/r0�statrF�get�st_mtime�appendr�st_size�lenr>�
create_commitr4r+r,rr)140rQ�relpath_to_abspath�prefix�files_to_upload�relpathrr��add_operations�commit_info�files141`         r&rjzCommitScheduler.push_to_hub�s����&�Y�$	�$	��$�0��%�%�'�'�'��L�L�H�I�I�I�"�"�"�"�"��$�)�)�&�1�1���"�"�"��<@�;L�T��)�/�/��4�4�7�7�7�7�RT�F�46�O�.�"�'�'�)�)�#�2� $� 4����
�
��1420��8�143�!���(�(���&�*�*�:�6�6�>��)�*�5���F�F�#�*�*�%�'1�)/�'�)9�'+�|�*.�-�	�������
�%$	�$	�$	�$	�$	�$	�$	�$	�$	�$	�$	����$	�$	�$	�$	�N����1�$�$��L�L�O�P�P�P��4�	���P�Q�Q�Q�144�145�#2�146�147�148��	���<�=�=�=��h�,�,��L��n�%�-��]�-�149�150��$�	E�	E�D�26�2D�D��t��/�/��s�EE�E�E�r6N)r6r()rrrrr!rrr"r#r�boolr
rrSrVrYr`rJrr[rrOrjr$r%r&r(r(#s������F�F�Z$%�&*�#'�"&�"&�#�:>�;?�$�$(�9=�A�A�A��A��3��9�%�	A�151�S�%�Z� �A��s�m�
A��C�=�A��3�-�A��$��A���}�A�!��t�C�y�#�~�!6�7�A�"�%��S�	�3��"7�8�A��A���!�A�#�8�B��H�#5�6�A� 152�!A�A�A�A�F����������������9��9�9�9�9��h�z�2�����&W�X�j�1�W�W�W�W�W�Wr%r(c���eZdZdZdeeefdeddfd�Zd�fd�Z	defd�Z153defd	�Zd154ef�fd�Zdefd�Z
efd
ededefd�Zddeedefd�Z�xZS)�
PartialFileIOaA file-like object that reads only the first part of a file.155 156    Useful to upload a file to the Hub when the user might still be appending data to it. Only the first part of the157    file is uploaded (i.e. the part that was available when the filesystem was first scanned).158 159    In practice, only used internally by the CommitScheduler to regularly push a folder to the Hub with minimal160    disturbance for the user. The object is passed to `CommitOperationAdd`.161 162    Only supports `read`, `tell` and `seek` methods.163 164    Args:165        file_path (`str` or `Path`):166            Path to the file to read.167        size_limit (`int`):168            The maximum number of bytes to read from the file. If the file is larger than this, only the first part169            will be read (and uploaded).170    �	file_pathrr6Nc���t|��|_|j�d��|_t	|tj|j�����j��|_	dS)N�rb)171r�172_file_path�open�_file�min�os�fstat�filenor��_size_limit)rQr�rs   r&rSzPartialFileIO.__init__OsY���y�/�/����_�)�)�$�/�/��173��z�2�8�D�J�4E�4E�4G�4G�+H�+H�+P�Q�Q����r%c�v��|j���t�����SrX)r��close�super�__del__)rQ�	__class__s �r&r�zPartialFileIO.__del__Ts+����174�������w�w��� � � r%c�(�d|j�d|j�d�S)Nz<PartialFileIO file_path=z size_limit=�>)r�r�rUs r&�__repr__zPartialFileIO.__repr__Xs"��X���X�X�T�EU�X�X�X�	175r%c��|jSrX)r�rUs r&�__len__zPartialFileIO.__len__]s����r%�namec���|�d��s|dvr!t���|��Std|�d����)N�_)�read�tell�seekz PartialFileIO does not support 'r9)�176startswithr��__getattribute__�NotImplementedError)rQr�r�s  �r&r�zPartialFileIO.__getattribute__`s\����?�?�3���	2�4�,177�$178�$179�180�7�7�+�+�D�1�1�1�!�"M�T�"M�"M�"M�N�N�Nr%c�4�|j���S)z!Return the current file position.)r�r�rUs r&r�zPartialFileIO.tellis���z��� � � r%�_PartialFileIO__offset�_PartialFileIO__whencec���|tkrt|��|z}t}|j�||��}||jkr|j�|j��S|S)z�Change the stream position to the given offset.181 182        Behavior is the same as a regular file, except that the position is capped to the size limit.183        )rr�rr�r�r�)rQr�r��poss    r&r�zPartialFileIO.seekmsc��184�x����4�y�y�8�+�H��H��j�o�o�h��1�1����!�!�!��:�?�?�4�#3�4�4�4��185r%������_PartialFileIO__sizec��|j���}|�|dkr|j|z186}nt||j|z187��}|j�|��S)z�Read at most `__size` bytes from the file.188 189        Behavior is the same as a regular file, except that it is capped to the size limit.190        Nr)r�r�r�r�r�)rQr��current�truncated_sizes    r&r�zPartialFileIO.read|s^��191�*�/�/�#�#���>�V�a�Z�Z�!�-��7�N�N�!���)9�G�)C�D�D�N��z���~�.�.�.r%r�)r�)rrrrrr!rr"rSr�r�r�r�r�rr�r�bytesr��
__classcell__)r�s@r&r�r�<s[���������$R�%��T�	�"2�R��R��R�R�R�R�192!�!�!�!�!�!�193�#�194�195�196�197�198 �� � � � �O�S�O�O�O�O�O�O�!�c�!�!�!�!�3;�
�
�S�
�C�
�s�
�
�
�
�/�/�8�C�=�/�%�/�/�/�/�/�/�/�/r%r�)$rM�loggingr�rd�concurrent.futuresr�dataclassesr�iorrr�pathlibr�	threadingr	r199�typingrrr
rr�huggingface_hub.hf_apirrrr�huggingface_hub.utilsr�	getLoggerrrHrr(r�r$r%r&�<module>r�s����
�
�
�����	�	�	�	�����%�%�%�%�%�%�!�!�!�!�!�!�*�*�*�*�*�*�*�*�*�*�������"�"�"�"�"�"�"�"�8�8�8�8�8�8�8�8�8�8�8�8�8�8�������������6�5�5�5�5�5�	��	�8�	$�	$����$��������������V�V�V�V�V�V�V�V�rL/�L/�L/�L/�L/�G�L/�L/�L/�L/�L/r%