Clyine1/phi3_image_question_generator

1---2library_name: transformers3tags: []4---5 6# Model Card for Model ID7 8A model purpose made for MCQ generation on cartoon images for lower primary language education.9Phi3-mini is used as the LM and CLIP is used as the vision encoder.10 11Requires a CUDA enabled devices to run.12Please run the model using Google Colab with a T4 card, or a local CUDA device with at least 10GB of available VRAM.13Ensure that CUDA is built according to your device requirements.14 15## Uses16 17The model is meant to be used with a custom pipeline. It returns a dict Object with with the following keys181. questions (str)192. choices (list)203. answer (str)214. desc (str)22 23```python24import requests25from transformers import AutoProcessor, LlavaForConditionalGeneration, Pipeline26from transformers.utils import PushToHubMixin27from PIL import Image28import random29import torch30 31class ImageToQuestionPipeline():32    def __init__(self, llava_model):33        self.processor = AutoProcessor.from_pretrained(llava_model)34        self.llava_model = LlavaForConditionalGeneration.from_pretrained(llava_model, torch_dtype=torch.float16).to(0)35 36    def __call__(self, image_path):37        commands = ["Generate a simple question\n","Suggest 1 correct answer\n","Suggest 3 incorrect answers\n", ""]38        prompt ='''39            <|user|>\n<image>\nDescribe this image in a passage\n<|end|>\n40            <|assistant|>\n41            '''42        image_file = image_path43        raw_image = Image.open(image_file)44        inputs = self.processor(prompt, raw_image, return_tensors='pt').to(0)45        artifacts = []46        while commands:47          inputs = self.processor(prompt, raw_image, return_tensors='pt').to(0)48          output = self.llava_model.generate(**inputs, eos_token_id=32007, max_new_tokens=500, do_sample=False)49          index = torch.where(output[0]==32001)[0][-1].item()50          text = self.processor.decode(output[0][index:], skip_special_tokens=True)51          artifacts.append(text)52          prompt += "{}<|end|>\n<|user|>\n{}<|end|>\n<|assistant|>\n".format(text,commands.pop(0)) 53 54        distractors = artifacts.pop(-1)55        a = distractors.split("\n")56        a = [x[3:] for x in a]57 58        correct_answer = random.randint(0,3)59        a.insert(correct_answer, artifacts[2])60 61        a = ["{}) {}".format(i+1, a[i]) for i in range(len(a))]62        answer = "Correct Answer: {}".format(correct_answer+1, a[correct_answer])63        result = {}64        result.update({"questions":artifacts[1]})65        result.update({"choices":a})66        result.update({"answer":answer})67        result.update({"desc":artifacts[0]})68        return result69 70 71pipe = ImageToQuestionPipeline("Clyine1/phi3_image_question_generator")72output = pipe(<image_file_path>)73print(json.dumps(output, indent=4))74 75"""76Generated output:77{78    "questions": "What is the color of the shirt the girl in the center of the image is wearing?",79    "choices": [80        "1) The girl in the center of the image is wearing a pink shirt.",81        "2) The girl in the center of the image is wearing a blue shirt.",82        "3) The girl in the center of the image is wearing a red shirt.",83        "4) The girl in the center of the image is wearing a green shirt."84    ],85    "answer": "Correct Answer: 1) The girl in the center of the image is wearing a pink shirt.",86    "desc": "The image depicts a lively scene at a playground. In the foreground, a young girl is sitting on a green slide, her face reflecting a sense of surprise or shock. She is dressed in a pink shirt and a red hat. Behind her, a boy is standing on the same slide, his arms crossed in a defensive posture. He is wearing a red shirt and a blue hat.\n\nIn the background, a girl is sitting on a swing, her legs swinging back and forth. She is wearing a pink shirt and a red hat. Another girl is standing on a blue slide, her arms crossed in a similar defensive posture as the boy on the green slide. She is wearing a blue shirt and a red hat.\n\nIn the distance, a boy is standing on a yellow slide, his arms crossed in a defensive posture. He is wearing a yellow shirt and a red hat. Another boy is standing on a blue slide, his arms crossed in a similar defensive posture as the boy on the yellow slide. He is wearing a blue shirt and a red hat.\n\nThe playground is surrounded by trees and buildings, providing a natural and urban backdrop to the scene. The colors of the playground equipment and the clothing of the children are vibrant and contrasting, adding to the lively atmosphere of the image."87}88"""89 90```