CoolFace
Modelpublic

hymenjj/llama-cpp-python-prebuilt

sourceHugging Faceupdated 7mo agoView on Hugging Face
0likes
settings.cpython-311.pyc37 linesDownload Raw Back to __pycache__
1200�iv!����ddlmZddlZddlmZmZmZmZmZm	Z	ddl3mZddlm
Z
mZddlmZddlZdejd<Gd	�d4e��ZGd�de��ZGd
�dee��ZGd�de��ZdS)�)�annotationsN)�Optional�List�Literal�Union�Dict�cast)�Self)�Field�model_validator)�BaseSettings��protected_namespacesc��eZdZUdZed���Zded<edd���Zd	ed5<eddd
���Zded<ee	j6d���Zded<eddd���Zded<edd���Z
ded<edd���Zded<ee	j��d���Zded<ee	j��d���Zded<edd ���Zd!ed"<edd#���Zd	ed$<ee	jd%���Zded&<ed'dd(���Zded)<ed*d+d,���Zded-<ed*d+d.���Zded/<eeej��d0zd+��d+d1���Zded2<eeej��d+��dd3���Zded4<ee	j�5��Z ded6<ed7d8���Z!d9ed:<ed7d;���Z"d9ed<<ed=�5��Z#d9ed><ed?�5��Z$d9ed@<edA�5��Z%d9edB<ed?�5��Z&d9edC<ed�5��Z'dedD<edEdF���Z(dedG<edEdH���Z)dedI<eddJ���Z*dedK<edEdL���Z+dedM<eddN���Z,dedO<edPddQ���Z-dedR<eddS���Z.d	edT<eddU���Z/d	edV<eddW���Z0dXedY<eddZ���Z1d	ed[<edd\���Z2d	ed]<edd^���Z3ded_<ed`da���Z4dbedc<eddde���Z5dedf<eddg���Z6d	edh<eddi���Z7d	edj<eddk���Z8d	edl<eddm���Z9d	edn<edodp���Z:dedq<eddr���Z;dsedt<eddu���Z<dsedv<edEdw���Z=dedx<e>dy�z��d~d}���Z?dS)�
ModelSettingsz*Model settings used to load a Llama model.z8The path to the model to use for generating completions.)�description�str�modelNz9The alias of the model to use for generating completions.��defaultr�
Optional[str]�model_aliasr�����z_The number of layers to put on the GPU. The rest will be on the CPU. Set -1 to move all to GPU.)r�ger�int�n_gpu_layerszThe split mode to use.�7split_modezMain GPU to use.�main_gpuz0Split layers across multiple GPUs in proportion.zOptional[List[float]]�tensor_splitFz&Whether to only return the vocabulary.�bool�8vocab_onlyz	Use mmap.�use_mmapz9Use mlock.�	use_mlockz�List of model kv overrides in the format key=type:value where type is one of (bool, int, float). Valid true values are (true, TRUE, 1), otherwise false.zOptional[List[str]]�kv_overridesz2comma seperated list of rpc servers for offloading�rpc_serverszRandom seed. -1 for random.�seedizThe context size.�n_ctxi�zThe batch size to use per eval.�n_batchz)The physical batch size used by llama.cpp�n_ubatch�z8The number of threads to use. Use -1 for max cpu threads�	n_threadszNThe number of threads to use when batch processing. Use -1 for max cpu threads�n_threads_batch)r�rope_scaling_typegzRoPE base frequency�float�rope_freq_basezRoPE frequency scaling factor�rope_freq_scaleg��yarn_ext_factorg�?�yarn_attn_factorg@@�yarn_beta_fast�yarn_beta_slow�
yarn_orig_ctxTz+if true, use experimental mul_mat_q kernels�	mul_mat_qzWhether to return logits.�10logits_allzWhether to use embeddings.�	embeddingz"Whether to offload kqv to the GPU.�offload_kqvzWhether to use flash attention.�11flash_attn�@z5Last n tokens to keep for repeat penalty calculation.�last_n_tokens_sizezoOptional path to base model, useful if using a quantized base model and you want to apply LoRA to an f16 model.�	lora_basez*Path to a LoRA file to apply to the model.�	lora_pathzEnable NUMA support.zUnion[bool, int]�numazChat format to use.�chat_formatz<Path to a CLIP model to use for multi-modal chat completion.�clip_model_pathz=Use a cache to reduce processing times for evaluated prompts.�cache�ramz5The type of cache to use. Only used if cache is True.zLiteral['ram', 'disk']�12cache_typelz;The size of the cache in bytes. Only used if cache is True.�13cache_sizez5The path to a HuggingFace tokenizer_config.json file.�hf_tokenizer_config_pathz~The model name or path to a pretrained HuggingFace tokenizer model. Same as you would pass to AutoTokenizer.from_pretrained().� hf_pretrained_model_name_or_pathz=The model repo id to use for the HuggingFace tokenizer model.�hf_model_repo_idzHMethod to use for speculative decoding. One of (prompt-lookup-decoding).�draft_model�14z2Number of tokens to predict using the draft model.�draft_model_num_pred_tokensz#Type of the key cache quantization.z
Optional[int]�type_kz%Type of the value cache quantization.�type_vz#Whether to print debug information.�verbose�before)�mode�returnr15c���tj��}tttt16f|��}|�dd��dkr||d<|�dd��dkr||d<|S)Nr,rrr-)�multiprocessing�	cpu_countr	rrr�get)�selfrU�valuess   �D/usr/local/lib/python3.11/site-packages/llama_cpp/server/settings.py�set_dynamic_defaultsz"ModelSettings.set_dynamic_defaults�sv��17$�-�/�/�	��d�3��8�n�d�+�+���:�:�k�1�%�%��+�+�"+�F�;���:�:�'��+�+�r�1�1�(1�F�$�%���)rRr18)@�__name__�19__module__�__qualname__�__doc__rr�__annotations__rr�	llama_cpp�LLAMA_SPLIT_MODE_LAYERrrrr!�llama_supports_mmapr"�llama_supports_mlockr#r$r%�LLAMA_DEFAULT_SEEDr&r'r)r*�maxrTrUr,r-�#LLAMA_ROPE_SCALING_TYPE_UNSPECIFIEDr.r0r1r2r3r4r5r6r7r8r9r:r;r=r>r?r@rArBrCrErFrGrHrIrJrLrMrNrOrrZrr[rYrrsJ	�������4�4���N����E�����"'���O�"�"�"�K�����20���
�u����L�����21�e��0�,����J������E���&����H�����22+0�%��F�+�+�+�L������u��#K����J������U�-�	�-�/�/�����H������e�.�	�.�0�0� ����I�����).���o�)�)�)�L�����"'���H�"�"�"�K�����23���,�:W����D�������t��7J�K�K�K�E�K�K�K�K��5���'H����G������E���'R����H������U���-�O�-�/�/�1�4�a�8�8��N����I�����24!�5���-�O�-�/�/��3�3��d����O�����25#�U��=���������"�E�#�;P�Q�Q�Q�N�Q�Q�Q�Q�"�U��!@����O�����#�U�4�0�0�0�O�0�0�0�0�#�e�C�0�0�0��0�0�0�0�!�E�$�/�/�/�N�/�/�/�/�!�E�#�.�.�.�N�.�.�.�.���q�)�)�)�M�)�)�)�)��e��"O����I������u�T�7R�S�S�S�J�S�S�S�S��e�E�7S�T�T�T�I�T�T�T�T����"F����K������u��#D����J�����$�e���K��������� %�u��F� � � �I����� %�u��@� � � �I�����26#�U��*����D�����27"'���)�"�"�"�K�����&+�U��R�&�&�&�O�����28�%��S����E�����*/���K�*�*�*�J������e��Q����J�����29/4�e��K�/�/�/������7<�e��U�7�7�7�$�����30',�e��S�'�'�'������31"'���^�"�"�"�K�����(-�u��H�(�(�(������32"�E��9����F�����"�E��;����F�����33�E��"G����G������_�
�����������r[rc�4�eZdZUdZedd���Zded<edd���Zd	ed34<edd���Zd
ed<edd���Z	d
ed<edd���Z35d
ed<edd���Zded<edd���Zded<edd���Z
ded<dS)�ServerSettingszAServer settings used to configure the FastAPI and Uvicorn server.�	localhostzListen addressrr�hosti@zListen portr�portNzSSL key file for HTTPSr�ssl_keyfilezSSL certificate file for HTTPS�ssl_certfilezIAPI key for authentication. If set all requests need to be authenticated.�api_keyTz=Whether to interrupt requests when a new request is received.r �interrupt_requestsFz;Disable EventSource pings (may be needed for some clients).�disable_ping_events�zIThe root path for the server. Useful when running behind a reverse proxy.�	root_path)r\r]r^r_rrkr`rlrmrnrorprqrsrr[rYriri�s��������K�K���k�7G�H�H�H�D�H�H�H�H���d�
�>�>�>�D�>�>�>�>�!&���":�"�"�"�K�����#(�%��"B�#�#�#�L�����#�U��_����G����� %�u��S� � � ������!&���Q�!�!�!�������U��_����I������r[ric��eZdZdS)�SettingsN)r\r]r^rr[rYruru�s�������Dr[ruc�8�eZdZUdZegd���Zded<dS)�ConfigFileSettingsz#Configuration file format settings.z
Model configsrzList[ModelSettings]�modelsN)r\r]r^r_rrxr`rr[rYrwrw�s8�������-�-�"'�%���"P�"P�"P�F�P�P�P�P�P�Pr[rw)�36__future__rrT�typingrrrrrr	�typing_extensionsr37�pydanticrr�pydantic_settingsr
ra�model_configrrirurwrr[rY�<module>rsr��"�"�"�"�"�"�����=�=�=�=�=�=�=�=�=�=�=�=�=�=�=�=�"�"�"�"�"�"�+�+�+�+�+�+�+�+�*�*�*�*�*�*�����57���0�1�v�v�v�v�v�L�v�v�v�r�����\����>	�	�	�	�	�~�}�	�	�	�Q�Q�Q�Q�Q��Q�Q�Q�Q�Qr[