abdeljalilELmajjodi/prompt-refinement-studio

<p align="center"> <img src="assets/logo.png" alt="Prompt Refinement Logo" width="250"/> </p> <div align="center"> <h1>Prompt Refinement </h1> </div> <div align="center">

    </div>

Iterative prompt engineering, solved.

Prompt Refinement is an automated framework that evolves your LLM prompts using a genetic-style optimization loop. It systematically improves prompt quality by minimizing cost and maximizing accuracy against your ground-truth dataset.

Instead of guessing what works, let Prompt Refinement treat your prompt as a hyperparameter to be optimized.

⚡️ Why Prompt Refinement?

Manual prompt engineering is tedious, unscientific, and hard to reproduce. Prompt Refinement automates the process:

• —🔄 Iterative Evolution: Uses a "Generator" LLM to strictly improve prompts based on specific performance bottlenecks.
• —🎯 Multi-Objective Scoring: Balances Accuracy, Novelty, and Cost to find the global optimum.
• —🤖 Mistral Native: Optimized for Mistral's model family (Medium/Large) for high-reasoning prompt generation.
• —📊 Traceable History: Tracks every prompt version, score, and latency for full observability.
• —🖥️ Streamlit UI: Includes a premium Mistral-branded web interface with live progress tracking.

🚀 Quick Start

1. Install

Prompt Refinement enables modern Python development with uv.

# Clone the repository
git clone https://github.com/yourusername/prompt-refinement.git
cd prompt-refinement

# Sync dependencies
uv sync

2. Configure Environment

Create a .env file with your Mistral API key:

echo "MISTRALAI_API_KEY=your_key_here" > .env

3. Run the Demo

Option A: Command Line

We include a baked-in example using the HaluEval dataset (hallucination detection):

uv run python prompt_refinement/cli/main.py

Output:

============================================================
OPTIMIZATION COMPLETE
============================================================
  Initial prompt : You are a helpful assistant.
  Final prompt   : Check: [Document] | [Summary] → y/n
  Initial score  : 0.7606
  Final score    : 0.8347
  Iterations     : 3
============================================================

Option B: Streamlit UI

Launch the interactive web interface:

uv run streamlit run prompt_refinement/ui/app.py

The UI provides:

• —Live progress tracking with real-time charts and iteration cards
• —Full sidebar configuration for models, weights, and dataset
• —Results dashboard with prompt comparison, score trajectory, and iteration details

🏗 Architecture

Prompt Refinement implements a feedback loop inspired by evolutionary algorithms. It doesn't just "rewrite" prompts; it measures them.

graph LR
    A[Initial Prompt] --> B(Executor)
    B --> C{Evaluator}
    C -->|Score & Bottleneck| D[Generator]
    D -->|New Variations| B

Components

Scoring System

The composite score is a weighted sum of three metrics, each normalized to [0, 1]:

⚖️ Optimizer Decision

The OptimizerDecision logic governs how the loop progresses and when it decides to finish.

1. Stop-Criteria (Early Stopping)

The orchestrator checks several conditions before proceeding to a new iteration:

• —Max Iterations: Loop terminates once max_iter_numb is reached.
• —Improvement Plateau: If the relative improvement of the score is less than the improvement_threshold (e.g., < 1%), the optimizer stops to save costs.
• —Convergence: If a variation reaches a "perfect" score threshold, the optimization is considered successful.

2. Best-Variant Selection

In each iteration, the PromptGenerator proposes multiple variations targeting the current Bottleneck. The optimizer then:

1. 1.Executes each variation against the full evaluation dataset.
2. 2.Calculates the composite score for every candidate.
3. 3.Selects the variation with the highest absolute score to become the "parent" for the next generation.

💻 Programmatic Usage

Integrate Prompt Refinement into your own evaluation pipeline:

from mistralai import Mistral
from datasets import load_dataset
from prompt_refinement import (
    EvaluatorConfig,
    OptimizerConfig,
    PromptOptimizer,
)

# 1. Setup Client
client = Mistral(api_key="...")

# 2. Load Data (Must have 'questions' and ground truth 'answers')
dataset = load_dataset("your_dataset", split="train[:50]")

# 3. Configure Optimization
config = OptimizerConfig(
    executor_model_name="mistral-medium-latest",  # Fast runner
    executor_client=client,
    generator_model_name="mistral-large-latest",  # Smart improver
    generator_client=client,
    max_iter_numb=5,
    improvement_threshold=0.01,
    questions="question_column",
    answers="answer_column",
    num_variations=3,
    task_description="Extract entities from financial reports...",
)

# 4. Run
optimizer = PromptOptimizer(config, EvaluatorConfig(), dataset)
result = optimizer.optimize("Extract info.")

print(f"Winner: {result.final_prompt}")

⚙️ Advanced Configuration

EvaluatorConfig

Fine-tune the evaluation logic:

OptimizerConfig

📁 Project Structure

prompt-refinement/
├── assets/                       # Branding & image assets
│   └── logo.png
├── notebooks/                    # Research & exploration notebooks
├── prompt_refinement/            # Main package
│   ├── core/                     # Internal logic (evaluators, optimizer, etc.)
│   ├── ui/                       # Streamlit web interface
│   │   └── app.py
│   ├── cli/                      # CLI entry points
│   │   └── main.py
│   └── __init__.py               # Public API exports
├── pyproject.toml                # Project metadata & dependencies
├── .streamlit/config.toml        # Streamlit theme configuration
└── tests/                        # Unit & integration tests

🤝 Contributing

Contributions are welcome! Please run the test suite before submitting a PR:

uv run pytest tests/