skyadmin/cog-webui-sd

1# Chill Watcher2consider deploy on:3- huggingface inference point4- replicate api5- lightning.ai6 7# platform comparison8> all support autoscaling9 10|platform|prediction speed|charges|deploy handiness|11|-|-|-|-|12|huggingface|fast:20s|high:$0.6/hr (without autoscaling)|easy:git push|13|replicate|fast if used frequently: 30s, slow if needs initialization: 5min|low: $0.02 per generation|difficult: build image and upload|14|lightning.ai|fast with app running: 20s, slow if idle: XXs|low: free $30 per month, $0.18 per init, $0.02 per run|easy: one command|15 16# platform deploy options17## huggingface18> [docs](https://huggingface.co/docs/inference-endpoints/guides/custom_handler)19 20- requirements: use pip packages in `requirements.txt`21- `init()` and `predict()` function: use `handler.py`, implement the `EndpointHandler` class22- more: modify `handler.py` for requests and inference and explore more highly-customized features23- deploy: git (lfs) push to huggingface repository(the whole directory including models and weights, etc.), and use inference endpoints to deploy. Click and deploy automaticly, very simple.24- call api: use the url provide by inference endpoints after endpoint is ready(build, initialize and in a "running" state), make a post request to the url using request schema definied in the `handler.py`25 26## replicate27> [docs](https://replicate.com/docs/guides/push-a-model)28 29- requirements: specify all requirements(pip packages, system packages, python version, cuda, etc.) in `cog.yaml`30- `init()` and `predict()` function: use `predict.py`, implement the `Predictor` class31- more: modify `predict.py`32- deploy: 33    1. get a linux GPU machine with 60GB disk space;34    2. install [cog](https://replicate.com/docs/guides/push-a-model) and [docker](https://docs.docker.com/engine/install/ubuntu/#set-up-the-repository)35    3. `git pull` the current repository from huggingface, including large model files36    4. after `predict.py` and `cog.yaml` is correctly coded, run `cog login`, `cog push`, then cog will build a docker image locally and push the image to replicate. As the image could take 30GB or so disk space, it would cost a lot network bandwidth.37- call api: if everything runs successfully and the docker image is pushed to replicate, you will see a web-ui and an API example directly in your replicate repository38 39## lightning.ai40> docs: [code](https://lightning.ai/docs/app/stable/levels/basic/real_lightning_component_implementations.html), [deploy](https://lightning.ai/docs/app/stable/workflows/run_app_on_cloud/)41 42- requirements: 43    - pip packages are listed in `requirements_lightning.txt`, because some requirements are different from those in huggingface. Rename it to `requirements.txt`44    - other pip packages, system packages and some big model weight files download commands, can be listed using a custom build config. Checkout `class CustomBuildConfig(BuildConfig)` in `app.py`. In a custom build config you can use many linux commands such as `wget` and `sudo apt-get update`. The custom build config will be executed on the `__init__()` of the `PythonServer` class45- `init()` and `predict()` function: use `app.py`, implement the `PythonServer` class. Note: 46    - some packages haven't been installed when the file is called(these packages may be installed when `__init__()` is called), so some import code should be in the function, not at the top of the file, or you may get import errors.47    - you can't save your own value to `PythonServer.self` unless it's predifined in the variables, so don't assign any self-defined variables to `self`48    - if you use the custom build config, you should implement `PythonServer`'s `__init()__` yourself, so don't forget to use the correct function signature49- more: ...50- deploy:51    - `pip install lightning`52    - prepare the directory on your local computer(no need to have a GPU)53    - list big files in the `.lightningignore` file to avoid big file upload and save deploy time cost54    - run `lightning run app app.py --cloud` in the local terminal, and it will upload the files in the directory to lightning cloud, and start deploying on the cloud55    - check error logs on the web-ui, use `all logs`56- call api: only if the app starts successfully, you can see a valid url in the `settings` page of the web-ui. Open that url, and you can see the api57 58### some stackoverflow:59install docker:60- https://docs.docker.com/engine/install/ubuntu/#set-up-the-repository61 62install git-lfs:63- https://github.com/git-lfs/git-lfs/blob/main/INSTALLING.md64linux:65```66curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | sudo bash67 68sudo apt-get install git-lfs69```70 71---72license: apache-2.073---74