hymenjj/llama-cpp-python-prebuilt
0
1�
2 00�i! � � � d Z ddlmZ ddlZddlZddlZddlZddlmZ ddl m3Z4m Z m Z m
Z
ddlmZmZ d� Zedk r e� � dS dS ) aY Example FastAPI server for llama.cpp.5 6To run this example:7 8```bash9pip install fastapi uvicorn sse-starlette pydantic-settings10export MODEL=../models/7B/...11```12 13Then run:14```15uvicorn llama_cpp.server.app:create_app --reload16```17 18or19 20```21python3 -m llama_cpp.server22```23 24Then visit http://localhost:8000/docs to see the interactive API docs.25 26� )� annotationsN)�27create_app)�Settings�ServerSettings�
ModelSettings�ConfigFileSettings)�add_args_from_model�parse_model_from_argsc 28 �* � d} t j | �� � }t |t � � |� dt29 d�� � d }g }|� � � } t j � d|j30 � � }|�r t j � |� � st d|� d�� � �t |d � � 5 }|� d31� � s|� d � � rCd d l}d d l}t% j |� |� |� � � � � � } n&t% j |� � � � � } t/ j | � � }| j }d d d � � n # 1 swxY w Y n+t5 t. |� � }t5 t6 |� � g}nZ# t8 $ rM}32t; |33t< j �
� � |� � � t= j! d� � Y d }34~35nd }36~37ww xY w|�J �|�J �tE ||�� � } tG j$ | t j% d|j& � � tO t j% d|j( � � � � |j) |j* �� � d S )Nu5 🦙 Llama.cpp python server. Host your own LLMs!🚀)� descriptionz
--config_filez Path to a config file to load.)�type�help� CONFIG_FILEz Config file z not found!�rbz.yamlz.ymlr )�file� )�server_settings�model_settings�HOST�PORT)�host�port� ssl_keyfile� ssl_certfile)+�argparse�ArgumentParserr r � add_argument�str�38parse_args�os�environ�get� config_file�path�exists�39ValueError�open�endswith�yaml�jsonr �model_validate_json�dumps� safe_load�readr �model_validate�modelsr40 r � Exception�print�sys�stderr�41print_help�exitr �uvicorn�run�getenvr �intr r r ) r �parserr r �argsr# �fr) r* �config_file_settings�e�apps �D/usr/local/lib/python3.11/site-packages/llama_cpp/server/__main__.py�mainrB + s� � �I�K�
�
$��
=�
=�
=�F����)�)�)�42����
�
-� � � � �43 .2�O�*,�N� � � � � �D���j�n�n�]�D�4D�E�E� � � J��7�>�>�+�.�.�
J� �!H� �!H�!H�!H�I�I�I��k�4�(�(�
=�A� �'�'��0�0� 44�K�4H�4H��4P�4P� 45� �K�K�K� �K�K�K�+=�+Q� �46�47�4�>�>�!�#4�#4�5�5�,� ,�(�(� ,>�+Q������,� ,�(� #1�"?�@T�"U�"U��!5�!<��
=�
=�
=�
=�
=�
=�
=�
=�
=�
=�
=����
=�
=�
=�
=�� 4�N�D� I� I�O� 3�M�4� H� H� I�N��� � � � �
�a�c�j�!�!�!�!������� ��� � � � � � � � ��������� � &� &� &� � %� %� %�48�'�%� � � �C� �K� �
�Y�v�� 3�
4�
4�
���6�?�#7�8�8�
9�
9�#�/�$�1� � � � � � s? �#A*G �
B/F�< G �F � G �F �/G � 49H�50AH�H�__main__)�__doc__�51__future__r r r3 r r7 �llama_cpp.server.appr �llama_cpp.server.settingsr r r r �llama_cpp.server.clir r52 rB �__name__� � rA �<module>rL s� ��� �0 #� "� "� "� "� "� � � � � 53�54�55�56� ���� ���� +� +� +� +� +� +�� � � � � � � � � � � � L� K� K� K� K� K� K� K�5� 5� 5�p �z����D�F�F�F�F�F� �rK