raulc0399/flux_dev_openpose_controlnet

1---2license: other3license_name: flux-1-dev-non-commercial-license4license_link: https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/LICENSE.5datasets:6- raulc0399/open_pose_controlnet7language:8- en9pipeline_tag: text-to-image10tags:11- Stable Diffusion12- image-generation13- Flux14- diffusers15- controlnet16---17 18# openpose controlnet for flux.dev19(big thanks to [oxen.ai](https://www.oxen.ai/) for sponsoring the GPU for the training)20 21## inference22 23an openpose controlnet for flux-dev, trained on https://huggingface.co/datasets/raulc0399/open_pose_controlnet24 25the controlnet model is trained for the xlabs ai pipeline https://github.com/XLabs-AI/x-flux26 27to install the pipeline, execute the following:28 29```30git clone https://github.com/XLabs-AI/x-flux.git31cd x-flux32python3 -m venv xflux_env33source xflux_env/bin/activate34pip install -r requirements.txt35```36 37to run the pipeline with controlnet:38 39```40python3 main.py \41 --prompt "person enjoying a day at the park, full hd, cinematic" \42 --image ~/open_pose_controlnet_dataset/validation_images/pose/3_pose_1024.jpg --control_type openpose \43 --local_path ./model.safetensors \44 --use_controlnet --model_type flux-dev \45 --width 1024 --height 1024  --timestep_to_start_cfg 2 \46 --num_steps 50 --true_gs 4 --guidance 4 \47 --save_path ~/gen_imgs48```49 50if the image has already been preprocessed comment out the line #146 from src/flux/xflux_pipeline.py51```52# self.annotator = Annotator(control_type, self.other_device)53```54 55## training56 57```58oxen clone https://hub.oxen.ai/raulc/open_pose_controlnet_dataset59git clone https://github.com/raulc0399/x-flux.git60cd x-flux61git checkout open_pose_training62python3 -m venv xflux_env63source xflux_env/bin/activate64pip install -r requirements.txt65huggingface-cli login66accelerate config67mkdir images68rsync -r ~/open_pose_controlnet_dataset/train/images/ images/69cp train_configs/test_openpose_controlnet.yaml train_configs/openpose_controlnet.yaml70accelerate launch train_flux_deepspeed_controlnet.py --config "train_configs/openpose_controlnet.yaml"71```72note 1: check the file train_configs/openpose_controlnet.yaml before starting73 74note 2: rsync is needed, cp does not work with that many files75 76note 3: the oxen repo has the caption files as json as expected by the training script77 78## results79 80using these 2 images:81 82![control image 1](https://huggingface.co/raulc0399/flux_dev_openpose_controlnet/resolve/main/2_pose_1024.jpg "control image 1" )83![control image 2](https://huggingface.co/raulc0399/flux_dev_openpose_controlnet/resolve/main/3_pose_1024.jpg "control image 2")84 85with these prompts:86 87"two friends sitting by each other enjoying a day at the park, full hd, cinematic"88"person enjoying a day at the park, full hd, cinematic"89 90resulted in these images:91 92![result image 1](https://huggingface.co/raulc0399/flux_dev_openpose_controlnet/resolve/main/prev_result_1_100.png "result image 1" )93![result image 2](https://huggingface.co/raulc0399/flux_dev_openpose_controlnet/resolve/main/prev_result_0_100.png "result image 2")94 95 96## License97 98Weights fall under the [FLUX.1 [dev]](https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/LICENSE.md) Non-Commercial License<br/>99