umair894/quickstart-trackio
0
1�
2 �1�h�< � �" � d dl Z d dlZd dlZd dlZd dlmZ d dlmZ d dlm Z m3Z4m Z d dl m
Z
d dlmZmZ d dlmZmZmZmZ d dlmZmZmZmZ d d lm Z ej: e � Z ed5� � G d � d
� � Z G d� d� Z! G d� de � Z"y)� N)�Future)� dataclass)�SEEK_END�SEEK_SET�BytesIO)�Path)�Lock�Thread)�Callable�Dict�List�Union)�DEFAULT_IGNORE_PATTERNS�6CommitInfo�CommitOperationAdd�HfApi)�filter_repo_objectsT)�frozenc �: � e Zd ZU dZeed< eed< eed< eed< y)�
_FileToUploadzWTemporary dataclass to store info about files to upload. Not meant to be used directly.�7local_path� path_in_repo�8size_limit�
last_modifiedN) �__name__�9__module__� __qualname__�__doc__r �__annotations__�str�int�float� � �C/usr/local/lib/python3.12/dist-packages/trackio/commit_scheduler.pyr r s � �a������O��r$ r c �& � e Zd ZdZdddddddddddd� dedeeef deeef d edz d10edz d edz d e dz d
edz de11e ez dz de12e ez dz de de dz de g df dz ddf d�Z
d d�Zd d�Zd d�Zd d�Zdefd�Zdedz fd�Zdedz fd�Zy) �CommitSchedulera�
13 Scheduler to upload a local folder to the Hub at regular intervals (e.g. push to hub every 5 minutes).14 15 The recommended way to use the scheduler is to use it as a context manager. This ensures that the scheduler is16 properly stopped and the last commit is triggered when the script ends. The scheduler can also be stopped manually17 with the `stop` method. Checkout the [upload guide](https://huggingface.co/docs/huggingface_hub/guides/upload#scheduled-uploads)18 to learn more about how to use it.19 20 Args:21 repo_id (`str`):22 The id of the repo to commit to.23 folder_path (`str` or `Path`):24 Path to the local folder to upload regularly.25 every (`int` or `float`, *optional*):26 The number of minutes between each commit. Defaults to 5 minutes.27 path_in_repo (`str`, *optional*):28 Relative path of the directory in the repo, for example: `"checkpoints/"`. Defaults to the root folder29 of the repository.30 repo_type (`str`, *optional*):31 The type of the repo to commit to. Defaults to `model`.32 revision (`str`, *optional*):33 The revision of the repo to commit to. Defaults to `main`.34 private (`bool`, *optional*):35 Whether to make the repo private. If `None` (default), the repo will be public unless the organization's default is private. This value is ignored if the repo already exists.36 token (`str`, *optional*):37 The token to use to commit to the repo. Defaults to the token saved on the machine.38 allow_patterns (`List[str]` or `str`, *optional*):39 If provided, only files matching at least one pattern are uploaded.40 ignore_patterns (`List[str]` or `str`, *optional*):41 If provided, files matching any of the patterns are not uploaded.42 squash_history (`bool`, *optional*):43 Whether to squash the history of the repo after each commit. Defaults to `False`. Squashing commits is44 useful to avoid degraded performances on the repo when it grows too large.45 hf_api (`HfApi`, *optional*):46 The [`HfApi`] client to use to commit to the Hub. Can be set with custom settings (user agent, token,...).47 on_before_commit (`Callable[[], None]`, *optional*):48 If specified, a function that will be called before the CommitScheduler lists files to create a commit.49 50 Example:51 ```py52 >>> from pathlib import Path53 >>> from huggingface_hub import CommitScheduler54 55 # Scheduler uploads every 10 minutes56 >>> csv_path = Path("watched_folder/data.csv")57 >>> CommitScheduler(repo_id="test_scheduler", repo_type="dataset", folder_path=csv_path.parent, every=10)58 59 >>> with csv_path.open("a") as f:60 ... f.write("first line")61 62 # Some time later (...)63 >>> with csv_path.open("a") as f:64 ... f.write("second line")65 ```66 67 Example using a context manager:68 ```py69 >>> from pathlib import Path70 >>> from huggingface_hub import CommitScheduler71 72 >>> with CommitScheduler(repo_id="test_scheduler", repo_type="dataset", folder_path="watched_folder", every=10) as scheduler:73 ... csv_path = Path("watched_folder/data.csv")74 ... with csv_path.open("a") as f:75 ... f.write("first line")76 ... (...)77 ... with csv_path.open("a") as f:78 ... f.write("second line")79 80 # Scheduler is now stopped and last commit have been triggered81 ```82 � NF) �everyr � repo_type�revision�private�token�allow_patterns�ignore_patterns�squash_history�hf_api�on_before_commit�repo_id� folder_pathr) r r* r+ r, r- r. r/ r0 r1 r2 �returnc
�� � | xs t |�� | _ |
| _ t |� j � j � | _ |xs d| _ | | _ |83�g }84nt |85t � r|86g}87|88t z | _ | j j � rt d| j � d�� �| j j dd�� | j j! |||d�� }|j" | _ || _ || _ || _ i | _ d | _ |dkD st d |� d�� �t/ � | _ || _ | | _ t6 j9 d89| j � d | j" � d | j2 � d
�� t; | j< d�� | _ | j> jA � tC jD | jF � d| _$ y )N)r- � z0'folder_path' must be a directory, not a file: '�'.T)�parents�exist_ok)r3 r, r* r: r z)'every' must be a positive integer, not 'zScheduled job to push 'z' to 'z' every z minutes.)�target�daemonF)%r �apir2 r �90expanduser�resolver4 r r. �91isinstancer r r/ �is_file�92ValueError�mkdir� create_repor3 r* r+ r- �
last_uploaded�last_push_timer �lockr) r0 �logger�infor93 �_run_scheduler�_scheduler_thread�start�atexit�register� _push_to_hub�_CommitScheduler__stopped)�selfr3 r4 r) r r* r+ r, r- r. r/ r0 r1 r2 �repo_urls r% �__init__zCommitScheduler.__init__l s� � �" �/�U�� /��� 0�� � � �,�7�7�9�A�A�C��� (� .�B��� ,��� � "� �O�
���
-� .� /�O� .�1H� H�� � � � � #� #� %� �B�4�CS�CS�BT�TV�W��
�
��� � �t�d� �;� �8�8�'�'��W� �D� (� 94�� �'�'�� �"��� ��
���95�02���,0����q�y� � H���r� R�S� S��F�� ���96� ,���� � �%�d�&6�&6�%7�v�d�l�l�^�8�TX�T^�T^�S_�_h� i� 97� "(�t�/B�/B�4�!P�� � � � �$�$�&�����)�)�*� ��r$ c � � d| _ y)ziStop the scheduler.98 99 A stopped scheduler cannot be restarted. Mostly for tests purposes.100 TN)rP �rQ s r% �stopzCommitScheduler.stop� s � �101 ��r$ c � � | S �Nr# rU s r% � __enter__zCommitScheduler.__enter__� s � �� r$ c �` � | j � j � | j � y rX )�trigger�resultrV )rQ �exc_type� exc_value� tracebacks r% �__exit__zCommitScheduler.__exit__� s � � � � �� � � � � � � �r$ c � � | j � | _ t j | j dz � | j102 ry�E)z7Dumb thread waiting between each scheduled push to Hub.�< N)r[ � last_future�time�sleepr) rP rU s r% rJ z CommitScheduler._run_scheduler� s6 � �� #�|�|�~�D� � �J�J�t�z�z�B�� '��~�~�� r$ c �L � | j j | j � S )z�Trigger a `push_to_hub` and return a future.103 104 This method is automatically called every `every` minutes. You can also call it manually to trigger a commit105 immediately, without waiting for the next scheduled commit.106 )r= �
run_as_futurerO rU s r% r[ zCommitScheduler.trigger� s � � �x�x�%�%�d�&7�&7�8�8r$ c �z � | j ry t j d� | j � }| j rQt j d� | j107 j
| j | j | j �� |S # t $ r }t j d|� �� � d }~ww xY w)Nz((Background) scheduled commit triggered.z$(Background) squashing repo history.)r3 r* �branchz Error while pushing to Hub: ) rP rH rI � push_to_hubr0 r= �super_squash_historyr3 r* r+ � Exception�error)rQ �value�es r% rO z CommitScheduler._push_to_hub� s� � � �>�>��� � �>�?� ��$�$�&�E��"�"�� � � B�C����-�-� �L�L�D�N�N�4�=�=� .� � �L��� � �L�L�.�q�c�2�
�
�� �s �A.B � B:� B5�5B:c 108 �X � | j 5 | j �| j � t j d� t | j109 j
d� � D �ci c]< }|j � r*|j | j110 � j � |��> }}| j r | j j d� � d�nd}g }t |j � | j | j �� D ]� }|| }|j! � }| j" j% |� � | j" | |j& k7 s�P|j) t+ |||z |j, |j& �� � �� ddd� t/ � dk( rt j d � yt j d111� |D �cg c]# }t1 |j2 |j � � ��% } }t j d � | j4 j7 | j8 | j: | d
| j< �� }112|D ]% } | j> | j" | j2 < �' tA j@ � | _! |113S c c}w # 1 sw Y ��xY wc c}w )a� 114 Push folder to the Hub and return the commit info.115 116 <Tip warning={true}>117 118 This method is not meant to be called directly. It is run in the background by the scheduler, respecting a119 queue mechanism to avoid concurrent commits. Making a direct call to the method might lead to concurrency120 issues.121 122 </Tip>123 124 The default behavior of `push_to_hub` is to assume an append-only folder. It lists all files in the folder and125 uploads only changed files. If no changes are found, the method returns without committing anything. If you want126 to change this behavior, you can inherit from [`CommitScheduler`] and override this method. This can be useful127 for example to compress data together in a single file before committing. For more details and examples, check128 out our [integration guide](https://huggingface.co/docs/huggingface_hub/main/en/guides/upload#scheduled-uploads).129 Nz-Listing files to upload for scheduled commit.z**/*�/r7 )r. r/ )r r r r r z4Dropping schedule commit: no changed file to upload.z9Removing unchanged files since previous scheduled commit.)�path_or_fileobjr z%Uploading files for scheduled commit.zScheduled Commit)r3 r* �130operations�commit_messager+ )"rG r2 rH �debug�sortedr4 �globrA � relative_to�as_posixr �stripr �keysr. r/ �statrE �get�st_mtime�appendr �st_size�lenr r r= �
create_commitr3 r* r+ r rd rF ) rQ �path�relpath_to_abspath�prefix�files_to_upload�relpathr r| �file_to_upload�add_operations� commit_info�files r% rj zCommitScheduler.push_to_hub� ss � �& �Y�Y��$�$�0��%�%�'� �L�L�H� I�131 #��$�$�)�)�&�1� �"� �D� �<�<�>� � � ��!1�!1�2�;�;�=�t�C� �
� "� <@�;L�;L��)�)�/�/��4�5�Q�7�RT�F� 46�O�.�"�'�'�)� #� 2� 2� $� 4� 4� ��132 0�� 8�133�!���(���&�&�*�*�:�6�>��)�)�*�5�� � �F�#�*�*�%�'1�)/�'�)9�'+�|�|�*.�-�-� �� �% �N �� �1� $� �L�L�O� P�� � � �P�Q� #2�134� #2��
� !/� 9� 9� +� 8� 8�
� #2� � 135� � � �<�=��h�h�,�,��L�L��n�n�%�-��]�]� -� 136� � $�D�26�2D�2D�D� � �t��� /� $� #�i�i�k������"� �Y��Z137s, �AJ�%AJ�&B/J�7J�(J'�J�J$�r5 N)r5 r' )r r r r r r r r! r" �bool�listr r rS rV rY r` rJ r r[ r rO rj r# r$ r% r' r' # sT � �F�Z $%�#'� $� #� #� �15�26� $� #�6:� > � �> � �3��9�%� > �138 �S�%�Z� � > � �D�j�
> � ��:�> � ��*�> � ���> � �T�z�> � �S� �C��$�.�> � �c��S� �4�/�> � �> � �� � > � #�2�t�8�,�t�3� > � 139�!> �@ ��� �9�� 9��j�4� /� �&Y�Z�$� .� Yr$ r' c � � � e Zd ZdZdeeef deddfd�Zd� fd� Z defd�Z140defd �Z d141ef� fd � Z defd �Z
efd
ededefd�Zddedz defd�Z� xZS )�
PartialFileIOa A file-like object that reads only the first part of a file.142 143 Useful to upload a file to the Hub when the user might still be appending data to it. Only the first part of the144 file is uploaded (i.e. the part that was available when the filesystem was first scanned).145 146 In practice, only used internally by the CommitScheduler to regularly push a folder to the Hub with minimal147 disturbance for the user. The object is passed to `CommitOperationAdd`.148 149 Only supports `read`, `tell` and `seek` methods.150 151 Args:152 file_path (`str` or `Path`):153 Path to the file to read.154 size_limit (`int`):155 The maximum number of bytes to read from the file. If the file is larger than this, only the first part156 will be read (and uploaded).157 � file_pathr r5 Nc �� � t |� | _ | j j d� | _ t |t j | j j � � j � | _ y )N�rb)158r �159_file_path�open�_file�min�os�fstat�filenor� � _size_limit)rQ r� r s r% rS zPartialFileIO.__init__N sN � � �y�/����_�_�)�)�$�/��160� �z�2�8�8�D�J�J�4E�4E�4G�+H�+P�+P�Q��r$ c �T �� | j j � t �| �
� S rX )r� �close�super�__del__)rQ � __class__s �r% r� zPartialFileIO.__del__S s �� � �161�162�����w�� � r$ c �< � d| j � d| j � d�S )Nz<PartialFileIO file_path=z size_limit=�>)r� r� rU s r% �__repr__zPartialFileIO.__repr__W s& � �'����'8� �T�EU�EU�DV�VW� X� 163r$ c � � | j S rX )r� rU s r% �__len__zPartialFileIO.__len__\ s � �� � � r$ �namec �j �� |j d� s|dv rt �| � |� S t d|� d�� �)N�_)�read�tell�seekz PartialFileIO does not support 'r8 )�164startswithr� �__getattribute__�NotImplementedError)rQ r� r� s �r% r� z PartialFileIO.__getattribute___ sB �� � �?�?�3� �4� ,165� $166�167 �7�+�D�1� 1�!�$D�T�F�"�"M�N�Nr$ c �6 � | j j � S )z!Return the current file position.)r� r� rU s r% r� zPartialFileIO.tellh s � ��z�z��� � r$ �_PartialFileIO__offset�_PartialFileIO__whencec �� � |t k( rt | � |z }t }| j j ||� }|| j168 kD r%| j j | j169 � S |S )z�Change the stream position to the given offset.170 171 Behavior is the same as a regular file, except that the position is capped to the size limit.172 )r r� r r� r� r� )rQ r� r� �poss r% r� zPartialFileIO.seekl s_ � �173 �x� ��4�y�8�+�H� �H��j�j�o�o�h��1�� ��!�!� !��:�:�?�?�4�#3�#3�4� 4��174r$ �_PartialFileIO__sizec �� � | j j � }|�|dk r| j |z175 }nt || j |z176 � }| j j |� S )z�Read at most `__size` bytes from the file.177 178 Behavior is the same as a regular file, except that it is capped to the size limit.179 r )r� r� r� r� r� )rQ r� �current�truncated_sizes r% r� zPartialFileIO.read{ s[ � �180 �*�*�/�/�#�� �>�V�a�Z� !� -� -�� 7�N� !���)9�)9�G�)C� D�N��z�z���~�.�.r$ r� )�����)r r r r r r r r! rS r� r� r� r� r� r r� �bytesr� �
__classcell__)r� s @r% r� r� ; s� �� ��$R�%��T� �"2� R�� R�� R�181!�182�#� 183�184 �� �O�S� O�!�c� !� 3;�
�S�
�C�
�s�
� /�3��:� /�u� /r$ r� )#rM �loggingr� rd �concurrent.futuresr � dataclassesr �ior r r �pathlibr � threadingr r185 �typingr r r
r �huggingface_hub.hf_apir r r r �huggingface_hub.utilsr � getLoggerr rH r r' r� r# r$ r% �<module>r� s� �� � � � � %� !� *� *� � "� .� .�� � 6� �� � �8� $�� �$��� � ��U� U�pL/�G� L/r$