Aluode/PerceptionLabPortable
0
1o
2 ��Yi" � @ s� d dl Z d dlZd dlZd dlZd dlZd dlmZ d dlmZm Z d dl3m Z d dl m
Z
d dlmZmZmZmZ dd� ZdaG d d4� d5e
�Zed g d ��G d
d� dej��ZG dd� de�ZdS )� N)�suppress)�cached_property�wraps)�parse_qs)�AbstractFileSystem)�"get_package_version_without_import�infer_storage_options� mirror_from�tokenizec s t � �� fdd��}|S )Nc 6 s^ z� | i |��W S t y. } z|js� |j^}} t|t�r)d|v r)ttj|�|�� d }~ww )Nzdoes not exist)�OSError�args�7isinstance�str�FileNotFoundError�errno�ENOENT)r �kwargs� exception�message��func� ��E:\DocsHouse\542 percep lab latest\PerceptionLab\PerceptionLab_Portable\python_embed\Lib\site-packages\fsspec/implementations/arrow.py�wrapper s 8��z wrap_exceptions.<locals>.wrapper)r )r r r r r �wrap_exceptions s
r c s� e Zd ZdZdZ� fdd�Zedd� �Zedd� �Z e9d d10� �Z d-d d
�Z dd� Z
dd� Zdd� Zedd� �Zedd� �Zedd� �Zed.dd ��Zed/d d ��Zed0d!d"��Zed-d#d$��Zed%d&� �Zed'd(� �Zd1� fd)d*� Z� fd+d,�Z� Z S )2�ArrowFSWrapperzwFSSpec-compatible wrapper of pyarrow.fs.FileSystem.11 12 Parameters13 ----------14 fs : pyarrow.fs.FileSystem15 16 �/c s$ t d�a|| _t� jdi |�� d S )NZpyarrowr )r �PYARROW_VERSION�fs�super�__init__)�selfr r �� __class__r r r 3 s zArrowFSWrapper.__init__c C s | j jS �N)r � type_name�r! r r r �protocol9 s zArrowFSWrapper.protocolc C s dt | jj| jj� S )NZhdfs_)r17 r �host�portr& r r r �fsid= s zArrowFSWrapper.fsidc C s* t |�}|d }|�d�r|dd � }|S )N�pathz//� )r �18startswith)�clsr+ �opsr r r �_strip_protocolA s19 20 z ArrowFSWrapper._strip_protocolFc sz � � |�}ddlm} z� fdd�� j�||��D �}W n ttfy1 � j|fi |��g}Y nw |r6|S dd� |D �S )Nr )� FileSelectorc s g | ]}� � |��qS r )� _make_entry��.0�entryr& r r �21<listcomp>O s ��z%ArrowFSWrapper.ls.<locals>.<listcomp>c S s g | ]}|d �qS )�namer r3 r r r r6 X s )r0 �22pyarrow.fsr1 r �
get_file_infor �NotADirectoryError�info)r! r+ �detailr r1 �entriesr r& r �lsJ s 23 2425��zArrowFSWrapper.lsc K s$ | � |�}| j�|g�\}| �|�S r$ )r0 r r9 r2 )r! r+ r r; r r r r; Z s 2627zArrowFSWrapper.infoc C s0 | � |�}z| �|� W dS ty Y dS w )NFT)r0 r; r �r! r+ r r r �exists_ s 28 ��zArrowFSWrapper.existsc C sl ddl m} |j|ju rd}n |j|ju rd}n|j|ju r*ttjt �29tj�|j ��d}|j |j ||j
d�S )Nr )�FileType� directory�file�other)r7 �size�type�mtime)r8 rA rF Z Directory�FileZNotFoundr r r �os�strerrorr+ rE rG )r! r; rA �kindr r r r2 h s �zArrowFSWrapper._make_entryc K s� | � |��d�}| � |��d�}| �|d��[}|� dt�d�� �}z%| �|d��}t�||� W d � n1 s9w Y | j� ||� W n" t30yh t t �� | j�
|� W d � � 1 sbw Y � w W d � d S 1 stw Y d S )Nr �rbz.tmp.� �wb)r0 �rstrip�_open�secretsZ token_hex�open�shutil� copyfileobjr �move�
BaseExceptionr r � delete_file)r! �path1�path2r ZlstreamZ tmp_fnameZrstreamr r r �cp_file{ s& � 3132����"�zArrowFSWrapper.cp_filec K s2 | � |��d�}| � |��d�}| j�||� d S )Nr )r0 rO r rU )r! rX rY r r r r �mv� s zArrowFSWrapper.mvc C � | � |�}| j�|� d S r$ )r0 r rW r? r r r �rm_file� � 33zArrowFSWrapper.rm_fileNc C sF | � |��d�}| �|�r|r| j�|� d S td��| j�|� d S )Nr z0Can't delete directories without recursive=False)r0 rO �isdirr �34delete_dir�35ValueErrorrW )r! r+ � recursiveZmaxdepthr r r �rm� s 36zArrowFSWrapper.rmrL Tc K s� |dkr|r | j j}n | j j}n|dkr| j j}n|dkr"| j j}ntd|����i }|dks1|s@tt�d�d �dkr@d |d< ||fi |��}t | ||||fi |��S ) NrL rN �abz'unsupported mode for Arrow filesystem: �.r � � compression)37r Zopen_input_fileZopen_input_streamZopen_output_streamZopen_append_streamra �intr �split� ArrowFile) r! r+ �mode�38block_size�seekabler �method�_kwargs�streamr r r rP � s 39404142 zArrowFSWrapper._openc K s4 | � |�}|r| j|dd� d S | jj|dd� d S )NT)�exist_okF�rb )r0 �makedirsr �43create_dir)r! r+ Zcreate_parentsr r r r �mkdir� s 44zArrowFSWrapper.mkdirc C s | � |�}| jj|dd� d S )NTrr )r0 r rt )r! r+ rq r r r rs � s 45zArrowFSWrapper.makedirsc C r\ r$ )r0 r r` r? r r r �rmdir� r^ zArrowFSWrapper.rmdirc C s | � |�}| j�|�jS r$ )r0 r r9 rG r? r r r �modified� s 46zArrowFSWrapper.modifiedc s* |� d|dv� t� j|fd d d�|��S )Nrm )Nr )�start�end)�47setdefaultr �cat_file)r! r+ rx ry r r" r r r{ � s zArrowFSWrapper.cat_filec s& |� dd� t� j||fi |�� d S )Nrm F)rz r �get_file)r! ZrpathZlpathr r" r r r| � s zArrowFSWrapper.get_file)F)FN)rL NT)T)NN) �__name__�48__module__� __qualname__�__doc__Z root_markerr �propertyr' r r* � classmethodr0 r> r; r@ r2 r rZ r[ r] rc rP ru rs rv rw r{ r| �
__classcell__r r r" r r ( sD 49505152 535455 56 5758r rp )�read�seek�tell�write�readable�writable�closerm c @ s2 e Zd Zd59dd�Zdd� Zedd� �Zdd � ZdS ) rj Nc K s. || _ || _|| _|| _| | _| _|| _d S r$ )r+ rk r rp � blocksizerl r )r! r rp r+ rk rl r r r r r � s 60zArrowFile.__init__c C s | S r$ r r& r r r � __enter__� s zArrowFile.__enter__c C s61 | j �� S r$ )rp rE r& r r r rE � s 62zArrowFile.sizec G s | � � S r$ )r� )r! r r r r �__exit__� s zArrowFile.__exit__r$ )r} r~ r r r� r� rE r� r r r r rj � s 636465 rj c s> e Zd ZdZdZ d � fdd� Zed d66� �Z� ZS ) �HadoopFileSystemz]A wrapper on top of the pyarrow.fs.HadoopFileSystem67 to connect it's interface with fsspecZhdfs�defaultr N� c 68 s: ddl m} |||||||d�} t� jdd| i|�� dS )a; 69 70 Parameters71 ----------72 host: str73 Hostname, IP or "default" to try to read from Hadoop config74 port: int75 Port to connect on, or default from Hadoop config if 076 user: str or None77 If given, connect as this username78 kerb_ticket: str or None79 If given, use this ticket for authentication80 replication: int81 set replication factor of file for write operations. default value is 3.82 extra_conf: None or dict83 Passed on to HadoopFileSystem84 r )r� )r( r) �user� kerb_ticket� replication�85extra_confr Nr )r8 r� r r )86r! r( r) r� r� r� r� r r� r r" r r r s �zHadoopFileSystem.__init__c C s� t | �}i }|�dd �r|d |d< |�dd �r |d |d< |�dd �r*|d |d< |�dd �rFt|d �}|�dd �rFt|d d �|d< |S )Nr( �usernamer� r) Z url_queryr� r )r �getr rh )r+ r/ �outZqueriesr r r �_get_kwargs_from_urls( s z&HadoopFileSystem._get_kwargs_from_urls)r� r NNr� N) r} r~ r r� r' r � staticmethodr� r� r r r" r r� � s �'r� )r �iorI rQ rS �87contextlibr � functoolsr r � urllib.parser Z fsspec.specr Z fsspec.utilsr r r r88 r r r �IOBaserj r� r r r r �<module> s( 1�