Modelpublic

prithivMLmods/JSONify-Flux

sourceHugging Faceapache-2.0updated 2y agoView on Hugging Face

3likes13downloads

Model Card

JSONify-Flux

The JSONify-Flux model is a fine-tuned version of Qwen2-VL, specifically tailored for Flux-generated image analysis, caption extraction, and structured JSON formatting. This model is optimized for tasks involving image-to-text conversion, Optical Character Recognition (OCR), and context-aware structured data extraction.

Key Enhancements:

—Advanced Image Understanding: JSONify-Flux has been trained using 30 million trainable parameters on Flux-generated images and their captions, ensuring precise image comprehension.

—Optimized for JSON Output: The model is designed to output structured JSON data, making it suitable for integration with databases, APIs, and automation pipelines.

—Enhanced OCR Capabilities: JSONify-Flux excels in recognizing and extracting text from images with a high degree of accuracy.

—Multimodal Processing: Supports both image and text inputs while generating structured JSON-formatted outputs.

—Multilingual Support: Trained to recognize text inside images in multiple languages, including English, Chinese, European languages, Japanese, Korean, Arabic, and more.

How to Use

python

from transformers import Qwen2VLForConditionalGeneration, AutoTokenizer, AutoProcessor
from qwen_vl_utils import process_vision_info

# Load the model with optimized parameters
model = Qwen2VLForConditionalGeneration.from_pretrained(
    "prithivMLmods/JSONify-Flux", torch_dtype="auto", device_map="auto"
)

# Recommended acceleration for performance optimization
# model = Qwen2VLForConditionalGeneration.from_pretrained(
#     "prithivMLmods/JSONify-Flux",
#     torch_dtype=torch.bfloat16,
#     attn_implementation="flash_attention_2",
#     device_map="auto",
# )

# Default processor
processor = AutoProcessor.from_pretrained("prithivMLmods/JSONify-Flux")

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "https://flux-generated.com/sample_image.jpeg",
            },
            {"type": "text", "text": "Extract structured information from this image in JSON format."},
        ],
    }
]

# Prepare for inference
text = processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
    text=[text],
    images=image_inputs,
    videos=video_inputs,
    padding=True,
    return_tensors="pt",
)
inputs = inputs.to("cuda")

# Generate output
generated_ids = model.generate(**inputs, max_new_tokens=256)
generated_ids_trimmed = [
    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)

JSON Output Example:

json

{
  "image_id": "sample_image.jpeg",
  "captions": [
    "A futuristic cityscape with neon lights.",
    "A digital artwork featuring an abstract environment."
  ],
  "recognized_text": "Welcome to Flux City!",
  "metadata": {
    "color_palette": ["#FF5733", "#33FF57", "#3357FF"],
    "detected_objects": ["building", "sign", "street light"]
  }
}

Key Features

1.Flux-Based Training Data
2.Trained using Flux-generated images and captions to ensure high-quality structured output.

1.Optical Character Recognition (OCR)
2.Extracts and processes textual content within images.

1.Structured JSON Output
2.Outputs information in JSON format for easy integration with various applications.

1.Conversational Capabilities
2.Handles multi-turn interactions with structured responses.

1.Image & Text Processing
2.Inputs can include images, text, or both, with JSON-formatted results.

1.Secure and Optimized Model Weights
2.Uses Safetensors for enhanced security and efficient model loading.