Modelpublic

yutengz/Action2Vision

sourceHugging Facemitupdated 1y agoView on Hugging Face

0likes12downloads

Model Card

Action2Vision: InstructPix2Pix Fine-tuning for Robotic Action Frame Prediction

GitHub: https://github.com/yutengzhang03/Action2Vision <img src='img/show-example.png'/>

Example

To use InstructPix2Pix, install diffusers using main for now. The pipeline will be available in the next release

bash

pip install diffusers accelerate safetensors transformers

python

import PIL
import requests
import torch
from diffusers import StableDiffusionInstructPix2PixPipeline, EulerAncestralDiscreteScheduler

model_id = "yutengz/Action2Vision"
pipe = StableDiffusionInstructPix2PixPipeline.from_pretrained(model_id, torch_dtype=torch.float16, safety_checker=None)
pipe.to("cuda")
pipe.scheduler = EulerAncestralDiscreteScheduler.from_config(pipe.scheduler.config)


to_tensor = transforms.ToTensor()
resize = transforms.Resize((256, 256))

def download_image(URL):
    return PIL.Image.open(requests.get(url, stream=True).raw).convert("RGB").resize((256, 256))

url = "https://github.com/yutengzhang03/Action2Vision/blob/main/img/source.png"
image = download_image(url)
prompt = "There is a hammer and a block in the middle of the table. If the block is closer to the left robotic arm, it uses the left arm to pick up the hammer and strike the block; otherwise, it does the opposite."
images = pipe(prompt, image=image).images
images[0]