Aluode/PerceptionLabPortable
0
1o
2 ��YiY? � @ s� d dl mZ d dlZd dlZd dlZd dlmZmZ d dlm Z d dl3m Z d dl m
Z
G dd� de�ZG d d4� d5e �ZG d d � d e
�ZdS )
� )� annotationsN)� HTTPAdapter�Retry)�override)�AbstractFileSystem)�AbstractBufferedFilec s" e Zd ZdZd� fdd� Z� ZS )�DatabricksExceptionz<6 Helper class for exceptions raised in this module.7 Nc s" t � �|� || _|| _|| _dS )z Create a new DatabricksExceptionN)�super�__init__�8error_code�message�details)�selfr r r
�� __class__� ��E:\DocsHouse\542 percep lab latest\PerceptionLab\PerceptionLab_Portable\python_embed\Lib\site-packages\fsspec/implementations/dbfs.pyr9 s 10z DatabricksException.__init__�N)�__name__�11__module__� __qualname__�__doc__r12 �
__classcell__r r r r r s r c s� e Zd ZdZ� fdd�Zed%dd��Zd&d d13�Zd&d d �Zd&d
d�Z d'dd�Z14 d(dd�Z d)dd�Z dd� Z
d&dd �Zd d � Zd d � Zd!d"� Zd*� fd#d$� Z� ZS )+�DatabricksFileSystemz�15 Get access to the Databricks filesystem implementation over HTTP.16 Can be used inside and outside of a databricks cluster.17 c sp || _ || _t�� | _tddg d�d�| _| j�dt| jd�� | jj �18dd| j� �i� t � j d19i |�� d S ) a� 20 Create a new DatabricksFileSystem.21 22 Parameters23 ----------24 instance: str25 The instance URL of the databricks cluster.26 For example for an Azure databricks cluster, this27 has the form adb-<some-number>.<two digits>.azuredatabricks.net.28 token: str29 Your personal token. Find out more30 here: https://docs.databricks.com/dev-tools/api/latest/authentication.html31 �32 g�������?)i� i� i� i� i� i� )�total�backoff_factor�status_forcelist�https://)� max_retries�
AuthorizationzBearer Nr )
�instance�token�requests�Session�sessionr �retries�mountr �headers�updater r33 )r r! r" �kwargsr r r r34 " s 35�z DatabricksFileSystem.__init__�return�!list[dict[str, str | int]] | Nonec C st | j �|�d�d� | �|�}|| j v r8| j | D ]}|d |�d�kr3|d dkr/|g S g S qt|��dS )z�Check cache for listing36 37 Returns listing, if found (may be empty list for a directory that38 exists but contains nothing), None if not in cache.39 �/N�name�type� directory)�dircache�pop�rstrip�_parent�FileNotFoundError)r �path�parent�entryr r r �_ls_from_cache> s 4041 42��z#DatabricksFileSystem._ls_from_cacheTc 43 K s� z| � |�}W n ty | j�| �|�d� d}Y nw |sVz | jddd|id�}W n tyC } z |jdkr>t|j�|�� d}~ww |� dg �}dd � |D �}|| j|< |rZ|S d44d � |D �S ) a" 45 List the contents of the given path.46 47 Parameters48 ----------49 path: str50 Absolute path51 detail: bool52 Return not only the list of filenames,53 but also additional information on file sizes54 and types.55 N�get�listr6 ��method�endpoint�json�RESOURCE_DOES_NOT_EXIST�filesc S s, g | ]}|d |d r
dnd|d d��qS )r6 �is_dirr0 �file� file_size)r. r/ �sizer ��.0�or r r �56<listcomp>q s ���z+DatabricksFileSystem.ls.<locals>.<listcomp>c S s g | ]}|d �qS )r. r rF r r r rI } s )57r9 r5 r1 r2 r4 � _send_to_apir r r r: )r r6 �detailr* �out�r�erA r r r �lsP s2
�5859�60 �� �61zDatabricksFileSystem.lsc 62 C s� |s-z| j ddd|id� td|� d��� ty, } z |jdkr" W Y d}~nd}~ww z | j d d63d|id� W n tyQ } z |jd krLt|j�|�� d}~ww | �| �|�� dS ) aE 64 Create a given absolute path and all of its parents.65 66 Parameters67 ----------68 path: str69 Absolute path to create70 exist_ok: bool71 If false, checks if the folder72 exists before creating it (and raises an73 Exception if this is the case)74 r: z75get-statusr6 r< zPath z already existsr@ N�post�mkdirs�RESOURCE_ALREADY_EXISTS)rJ �FileExistsErrorr r r �invalidate_cacher4 )r r6 �exist_okrN r r r �makedirs s(
76�77��78 ��z DatabricksFileSystem.makedirsc K s |st �| j|fi |�� dS )a! 79 Create a given absolute path and all of its parents.80 81 Parameters82 ----------83 path: str84 Absolute path to create85 create_parents: bool86 Whether to create all parents or not.87 "False" is not implemented so far.88 N)�NotImplementedErrorrQ )r r6 Zcreate_parentsr* r r r �mkdir� s zDatabricksFileSystem.mkdirFc 89 K sz z
| j dd||d�d� W n% ty2 } z|jdkr"| j||d� � |jdkr-t|j�|�� d}~ww | �| �|�� dS )z�90 Remove the file or folder at the given absolute path.91 92 Parameters93 ----------94 path: str95 Absolute path what to remove96 recursive: bool97 Recursively delete all files in a folder.98 rP �delete)r6 � recursiver< ZPARTIAL_DELETEZIO_ERRORN)rJ r r �rm�OSErrorr rT r4 )r r6 rZ r* rN r r r r[ � s 99�100101� ��102zDatabricksFileSystem.rmNc 103 K s� |rt �|rt �z
| jdd||d�d� W n# ty8 } z|jdkr(t|j�|�|jdkr3t|j�|�� d}~ww | �| �|�� | �| �|�� dS )a� 104 Move a source to a destination path.105 106 A note from the original [databricks API manual]107 (https://docs.databricks.com/dev-tools/api/latest/dbfs.html#move).108 109 When moving a large number of files the API call will time out after110 approximately 60s, potentially resulting in partially moved data.111 Therefore, for operations that move more than 10k files, we strongly112 discourage using the DBFS REST API.113 114 Parameters115 ----------116 source_path: str117 From where to move (absolute path)118 destination_path: str119 To where to move (absolute path)120 recursive: bool121 Not implemented to far.122 maxdepth:123 Not implemented to far.124 rP �move)� source_path�destination_pathr< r@ rR N) rW rJ r r r5 r rS rT r4 )r r^ r_ rZ Zmaxdepthr* rN r r r �mv� s( 125�126 127 ��zDatabricksFileSystem.mv�rb�defaultc K s t | |f||d�|��S )z�128 Overwrite the base class method to make sure to create a DBFile.129 All arguments are copied from the base method.130 131 Only the default blocksize is allowed.132 )�mode�133block_size)�DatabricksFile)r r6 rc rd r* r r r �_open� s zDatabricksFileSystem._openc C s� |dkr | j j}n|dkr| j j}ntd|� ���tj�d| j� d�|�}|||d�}z |�� W |� � S t j134y[ } zz|j � � }W n135 t
yN |d�w tdi |��|�d}~ww ) a� 136 Send the given json to the DBFS API137 using a get or post request (specified by the argument `method`).138 139 Parameters140 ----------141 method: str142 Which http method to use for communication; "get" or "post".143 endpoint: str144 Where to send the request to (last part of the API URL)145 json: dict146 Dictionary of information to send147 rP r: zDo not understand method r z/api/2.0/dbfs/)r? Nr )r% rP r: �148ValueError�urllib�parse�urljoinr! �raise_for_statusr# � HTTPError�responser? � Exceptionr ) r r= r> r? Z session_call�urlrM rN Zexception_jsonr r r rJ s& 149150 151 � ���z!DatabricksFileSystem._send_to_apic 152 C sR z| j dd||d�d�}|d W S ty( } z |jdkr#t|j�|�� d}~ww )a{ 153 Internal function to create a handle, which can be used to154 write blocks of a file to DBFS.155 A handle has a unique identifier which needs to be passed156 whenever written during this transaction.157 The handle is active for 10 minutes - after that a new158 write transaction needs to be created.159 Make sure to close the handle after you are finished.160 161 Parameters162 ----------163 path: str164 Absolute path for this file.165 overwrite: bool166 If a file already exist at this location, either overwrite167 it or raise an exception.168 rP �create)r6 � overwriter< �handlerR N)rJ r r rS r )r r6 rq rM rN r r r �_create_handle. s �169170 ��z#DatabricksFileSystem._create_handlec 171 C sL z
| j ddd|id� W dS ty% } z |jdkr t|j�|�� d}~ww )z�172 Close a handle, which was opened by :func:`_create_handle`.173 174 Parameters175 ----------176 handle: str177 Which handle to close.178 rP �closerr r< r@ N)rJ r r r5 r )r rr rN r r r �
_close_handleM s 179 ��z"DatabricksFileSystem._close_handlec 180 C sr t �|��� }z| jdd||d�d� W dS ty8 } z|jdkr(t|j�|�|jdkr3t|j�|�� d}~ww )a� 181 Upload data to an already opened file handle182 (opened by :func:`_create_handle`).183 The maximal allowed data size is 1MB after184 conversion to base64.185 Remember to close the handle when you are finished.186 187 Parameters188 ----------189 handle: str190 Which handle to upload data to.191 data: bytes192 Block of data to add to the handle.193 rP z add-block�rr �datar< r@ ZMAX_BLOCK_SIZE_EXCEEDEDN) �base64� b64encode�decoderJ r r r5 r rg )r rr rw rN r r r � _add_data^ s �194 195 ��z DatabricksFileSystem._add_datac 196 C st z| j dd|||| d�d�}t�|d �W S ty9 } z|jdkr)t|j�|�|jdv r4t|j�|�� d}~ww ) a� 197 Download data in bytes from a given absolute path in a block198 from [start, start+length].199 The maximum number of allowed bytes to read is 1MB.200 201 Parameters202 ----------203 path: str204 Absolute path to download data from205 start: int206 Start position of the block207 end: int208 End position of the block209 r: �read)r6 �offset�lengthr< rw r@ )ZINVALID_PARAMETER_VALUEZMAX_READ_SIZE_EXCEEDEDN)rJ rx � b64decoder r r5 r rg )r r6 �start�endrM rN r r r � _get_data| s �210 211 ��z DatabricksFileSystem._get_datac s2 |d u r212| j �� n| j �|d � t� �|� d S r )r1 �clearr2 r rT )r r6 r r r rT � s z%DatabricksFileSystem.invalidate_cache)r+ r, )T�F)FN)ra rb r )r r r r r213 r r9 rO rV rX r[ r` rf rJ rs ru r{ r� rT r r r r r r s$ 214215/216!217 218�219. 220) r c sT e Zd ZdZdZ d� fdd � Zd221d � Zdd
d�Zdd� Zddd�Z � Z222S )re zH223 Helper class for files referenced in the DatabricksFileSystem.224 i ra rb T� readaheadNc sX |du s|dkr | j }|| j ksJ d|� ���t� j||f|||||p$i d�|�� dS )zr225 Create a new instance of the DatabricksFile.226 227 The blocksize needs to be the default one.228 Nrb z,Only the default block size is allowed, not )rc rd �229autocommit�230cache_type�
cache_options)�DEFAULT_BLOCK_SIZEr r231 ) r �fsr6 rc rd r� r� r� r* r r r r232 � s" ���233�zDatabricksFile.__init__c C s | j �| j�| _dS )z(Internal function to start a file uploadN)r� rs r6 rr )r r r r �_initiate_upload� s z DatabricksFile._initiate_uploadFc sj | j �d� | j �� � � fdd�| �t� ��D �}|D ] }| jj| j|d� q|r3| jj| jd� dS dS )z<Internal function to add a chunk of data to a started uploadr c s g | ]234\}}� ||� �qS r r )rG r� r� �rw r r rI � s �z0DatabricksFile._upload_chunk.<locals>.<listcomp>rv )rr TN) �buffer�seek�getvalue�_to_sized_blocks�lenr� r{ rr ru )r �finalZ data_chunks�235data_chunkr r� r �
_upload_chunk� s 236237 ��z DatabricksFile._upload_chunkc C s> d}|| }| � ||�D ]\}}|| jj| j||d�7 }q |S )z-Internal function to download a block of data� )r6 r� r� )r� r� r� r6 )r r� r� Z
return_bufferr~ Z chunk_startZ chunk_endr r r � _fetch_range� s 238�zDatabricksFile._fetch_ranger c c s@ � || }t ||| j�D ]}|}t||| j �}||fV q dS )zGHelper function to split a range from 0 to total_length into blocksizesN)�range� blocksize�min)r r~ r� r� r� Z239data_startZdata_endr r r r� � s � �z DatabricksFile._to_sized_blocks)ra rb Tr� Nr� )r ) r r r r r� r240 r� r� r� r� r r r r r re � s �"241 re )�242__future__r rx rh r# Zrequests.adaptersr r Ztyping_extensionsr Zfsspecr Z fsspec.specr rn r r re r r r r �<module> s