DevMastersZA/Marco_Professional_Profile

1# Multi-Model Evaluator (2_lab2.py)2 3A Python script that evaluates and compares the performance of multiple AI language models by generating a challenging question, collecting responses from various providers, and ranking them using a judge model.4 5## Overview6 7This script performs the following steps:81. **Question Generation**: Uses an OpenAI model to generate a challenging, real-world question92. **Multi-Model Evaluation**: Sends the question to multiple AI models from different providers103. **Response Collection**: Gathers and displays all responses with timing information114. **Judging**: Uses a judge model to rank the responses based on correctness, depth, clarity, and helpfulness12 13## Prerequisites14 15- Python 3.7 or higher16- API keys for the AI providers you want to test (at minimum, OpenAI API key is required)17 18## Installation19 201. **Install required Python packages:**21 22```bash23pip install openai anthropic python-dotenv24```25 26Or if you have a requirements file:27 28```bash29pip install -r requirements.txt30```31 32Required packages:33- `openai` - For OpenAI API calls and OpenAI-compatible APIs34- `anthropic` - For Anthropic/Claude API calls35- `python-dotenv` - For loading environment variables from `.env` file36 37## Environment Setup38 391. **Create a `.env` file** in the same directory as `2_lab2.py` (or in the project root)40 412. **Add your API keys** to the `.env` file:42 43```env44# Required45OPENAI_API_KEY=your_openai_api_key_here46 47# Optional (add only if you want to test these providers)48ANTHROPIC_API_KEY=your_anthropic_api_key_here49GOOGLE_API_KEY=your_google_api_key_here50DEEPSEEK_API_KEY=your_deepseek_api_key_here51GROQ_API_KEY=your_groq_api_key_here52OLLAMA_BASE_URL=http://localhost:1143453```54 55**Note:** Only `OPENAI_API_KEY` is strictly required. The script will skip providers for which API keys are missing.56 57## Supported Models58 59The script is configured to test the following models (you can modify the `COMPETITORS` list in the script):60 61- **Claude Sonnet 4.5** (Anthropic) - Requires `ANTHROPIC_API_KEY`62- **GPT-5 Nano** (OpenAI) - Requires `OPENAI_API_KEY`63- **Gemini 2.0 Flash** (Google) - Requires `GOOGLE_API_KEY`64- **Llama 3.2** (via Ollama) - Requires `OLLAMA_BASE_URL` pointing to local Ollama instance65- **DeepSeek Chat** (DeepSeek) - Requires `DEEPSEEK_API_KEY`66- **GPT-OSS-120B** (via Groq) - Requires `GROQ_API_KEY`67 68## Usage69 701. **Ensure your `.env` file is set up** with at least the `OPENAI_API_KEY`71 722. **Run the script:**73 74```bash75python 2_lab2.py76```77 78The script will:79- Generate a challenging question80- Display the question81- Query each configured model (skipping those without API keys)82- Display each response with timing information83- Use a judge model to rank all responses84- Display the final rankings with scores and justifications85 86## Customization87 88You can customize the script by modifying:89 90- **`QUESTION_GENERATOR_MODEL`** (line 167): The model used to generate questions (default: `"gpt-4.1-mini"`)91- **`JUDGE_MODEL`** (line 319): The model used to judge responses (default: `"o3-mini"`)92- **`COMPETITORS`** list (lines 196-227): Add, remove, or modify the models to test93 94## Notes95 96- Models without corresponding API keys will be skipped gracefully97- The script uses OpenAI's Responses API for some models and standard Chat Completions API for others98- Ollama requires a local instance running and accessible at the `OLLAMA_BASE_URL`99- Response times are measured and displayed for each model100- The judge model outputs JSON-formatted rankings with scores (0-10) and justifications101 102## Troubleshooting103 104- **"OPENAI_API_KEY is required"**: Make sure your `.env` file contains a valid OpenAI API key105- **"ANTHROPIC_API_KEY missing"**: This is expected if you don't have an Anthropic key. The script will skip Anthropic models106- **Ollama connection errors**: Ensure Ollama is running locally and accessible at the configured `OLLAMA_BASE_URL`107- **Import errors**: Make sure all required packages are installed: `pip install openai anthropic python-dotenv`108 109