DevMastersZA/Marco_Professional_Profile
0
1{2 "cells": [3 {4 "cell_type": "code",5 "execution_count": null,6 "metadata": {},7 "outputs": [],8 "source": [9 "# If you don't know what any of these packages do - you can always ask ChatGPT for a guide!\n",10 "from dotenv import load_dotenv\n",11 "from openai import OpenAI\n",12 "from pypdf import PdfReader\n",13 "import gradio as gr"14 ]15 },16 {17 "cell_type": "code",18 "execution_count": null,19 "metadata": {},20 "outputs": [],21 "source": [22 "load_dotenv(override=True)\n",23 "openai = OpenAI()"24 ]25 },26 {27 "cell_type": "code",28 "execution_count": null,29 "metadata": {},30 "outputs": [],31 "source": [32 "reader = PdfReader(\"me/linkedin.pdf\")\n",33 "linkedin = \"\"\n",34 "for page in reader.pages:\n",35 " text = page.extract_text()\n",36 " if text:\n",37 " linkedin += text"38 ]39 },40 {41 "cell_type": "code",42 "execution_count": null,43 "metadata": {},44 "outputs": [],45 "source": [46 "print(linkedin)"47 ]48 },49 {50 "cell_type": "code",51 "execution_count": null,52 "metadata": {},53 "outputs": [],54 "source": [55 "with open(\"me/summary.txt\", \"r\", encoding=\"utf-8\") as f:\n",56 " summary = f.read()"57 ]58 },59 {60 "cell_type": "code",61 "execution_count": null,62 "metadata": {},63 "outputs": [],64 "source": [65 "name = \"Ed Donner\""66 ]67 },68 {69 "cell_type": "code",70 "execution_count": null,71 "metadata": {},72 "outputs": [],73 "source": [74 "system_prompt = f\"You are acting as {name}. You are answering questions on {name}'s website, \\\n",75 "particularly questions related to {name}'s career, background, skills and experience. \\\n",76 "Your responsibility is to represent {name} for interactions on the website as faithfully as possible. \\\n",77 "You are given a summary of {name}'s background and LinkedIn profile which you can use to answer questions. \\\n",78 "Be professional and engaging, as if talking to a potential client or future employer who came across the website. \\\n",79 "If you don't know the answer, say so.\"\n",80 "\n",81 "system_prompt += f\"\\n\\n## Summary:\\n{summary}\\n\\n## LinkedIn Profile:\\n{linkedin}\\n\\n\"\n",82 "system_prompt += f\"With this context, please chat with the user, always staying in character as {name}.\"\n"83 ]84 },85 {86 "cell_type": "code",87 "execution_count": null,88 "metadata": {},89 "outputs": [],90 "source": [91 "print(system_prompt)"92 ]93 },94 {95 "cell_type": "code",96 "execution_count": null,97 "metadata": {},98 "outputs": [],99 "source": [100 "def chat(message, history):\n",101 " messages = [{\"role\": \"system\", \"content\": system_prompt}] + history + [{\"role\": \"user\", \"content\": message}]\n",102 " response = openai.chat.completions.create(model=\"gpt-4o-mini\", messages=messages)\n",103 " return response.choices[0].message.content"104 ]105 },106 {107 "cell_type": "markdown",108 "metadata": {},109 "source": [110 "## Special note for people not using OpenAI\n",111 "\n",112 "Some providers, like Groq, might give an error when you send your second message in the chat.\n",113 "\n",114 "This is because Gradio shoves some extra fields into the history object. OpenAI doesn't mind; but some other models complain.\n",115 "\n",116 "If this happens, the solution is to add this first line to the chat() function above. It cleans up the history variable:\n",117 "\n",118 "```python\n",119 "history = [{\"role\": h[\"role\"], \"content\": h[\"content\"]} for h in history]\n",120 "```\n",121 "\n",122 "You may need to add this in other chat() callback functions in the future, too."123 ]124 },125 {126 "cell_type": "code",127 "execution_count": null,128 "metadata": {},129 "outputs": [],130 "source": [131 "gr.ChatInterface(chat, type=\"messages\").launch()"132 ]133 },134 {135 "cell_type": "markdown",136 "metadata": {},137 "source": [138 "## A lot is about to happen...\n",139 "\n",140 "1. Be able to ask an LLM to evaluate an answer\n",141 "2. Be able to rerun if the answer fails evaluation\n",142 "3. Put this together into 1 workflow\n",143 "\n",144 "All without any Agentic framework!"145 ]146 },147 {148 "cell_type": "code",149 "execution_count": null,150 "metadata": {},151 "outputs": [],152 "source": [153 "# Create a Pydantic model for the Evaluation\n",154 "\n",155 "from pydantic import BaseModel\n",156 "\n",157 "class Evaluation(BaseModel):\n",158 " is_acceptable: bool\n",159 " feedback: str"160 ]161 },162 {163 "cell_type": "code",164 "execution_count": null,165 "metadata": {},166 "outputs": [],167 "source": [168 "evaluator_system_prompt = f\"You are an evaluator that decides whether a response to a question is acceptable. \\\n",169 "You are provided with a conversation between a User and an Agent. Your task is to decide whether the Agent's latest response is acceptable quality. \\\n",170 "The Agent is playing the role of {name} and is representing {name} on their website. \\\n",171 "The Agent has been instructed to be professional and engaging, as if talking to a potential client or future employer who came across the website. \\\n",172 "The Agent has been provided with context on {name} in the form of their summary and LinkedIn details. Here's the information:\"\n",173 "\n",174 "evaluator_system_prompt += f\"\\n\\n## Summary:\\n{summary}\\n\\n## LinkedIn Profile:\\n{linkedin}\\n\\n\"\n",175 "evaluator_system_prompt += f\"With this context, please evaluate the latest response, replying with whether the response is acceptable and your feedback.\""176 ]177 },178 {179 "cell_type": "code",180 "execution_count": null,181 "metadata": {},182 "outputs": [],183 "source": [184 "def evaluator_user_prompt(reply, message, history):\n",185 " user_prompt = f\"Here's the conversation between the User and the Agent: \\n\\n{history}\\n\\n\"\n",186 " user_prompt += f\"Here's the latest message from the User: \\n\\n{message}\\n\\n\"\n",187 " user_prompt += f\"Here's the latest response from the Agent: \\n\\n{reply}\\n\\n\"\n",188 " user_prompt += \"Please evaluate the response, replying with whether it is acceptable and your feedback.\"\n",189 " return user_prompt"190 ]191 },192 {193 "cell_type": "code",194 "execution_count": null,195 "metadata": {},196 "outputs": [],197 "source": [198 "import os\n",199 "gemini = OpenAI(\n",200 " api_key=os.getenv(\"GOOGLE_API_KEY\"), \n",201 " base_url=\"https://generativelanguage.googleapis.com/v1beta/openai/\"\n",202 ")"203 ]204 },205 {206 "cell_type": "code",207 "execution_count": null,208 "metadata": {},209 "outputs": [],210 "source": [211 "def evaluate(reply, message, history) -> Evaluation:\n",212 "\n",213 " messages = [{\"role\": \"system\", \"content\": evaluator_system_prompt}] + [{\"role\": \"user\", \"content\": evaluator_user_prompt(reply, message, history)}]\n",214 " response = gemini.beta.chat.completions.parse(model=\"gemini-2.5-flash\", messages=messages, response_format=Evaluation)\n",215 " return response.choices[0].message.parsed"216 ]217 },218 {219 "cell_type": "code",220 "execution_count": null,221 "metadata": {},222 "outputs": [],223 "source": [224 "messages = [{\"role\": \"system\", \"content\": system_prompt}] + [{\"role\": \"user\", \"content\": \"do you hold a patent?\"}]\n",225 "response = openai.chat.completions.create(model=\"gpt-4o-mini\", messages=messages)\n",226 "reply = response.choices[0].message.content"227 ]228 },229 {230 "cell_type": "code",231 "execution_count": null,232 "metadata": {},233 "outputs": [],234 "source": [235 "reply"236 ]237 },238 {239 "cell_type": "code",240 "execution_count": null,241 "metadata": {},242 "outputs": [],243 "source": [244 "evaluate(reply, \"do you hold a patent?\", messages[:1])"245 ]246 },247 {248 "cell_type": "code",249 "execution_count": null,250 "metadata": {},251 "outputs": [],252 "source": [253 "def rerun(reply, message, history, feedback):\n",254 " updated_system_prompt = system_prompt + \"\\n\\n## Previous answer rejected\\nYou just tried to reply, but the quality control rejected your reply\\n\"\n",255 " updated_system_prompt += f\"## Your attempted answer:\\n{reply}\\n\\n\"\n",256 " updated_system_prompt += f\"## Reason for rejection:\\n{feedback}\\n\\n\"\n",257 " messages = [{\"role\": \"system\", \"content\": updated_system_prompt}] + history + [{\"role\": \"user\", \"content\": message}]\n",258 " response = openai.chat.completions.create(model=\"gpt-4o-mini\", messages=messages)\n",259 " return response.choices[0].message.content"260 ]261 },262 {263 "cell_type": "markdown",264 "metadata": {},265 "source": [266 "## Improved Workflow: Iterative Reflection Loop\n",267 "\n",268 "Previous version's `chat()` function calls `rerun()` **at most once** and then returns the result unconditionally — the rerun answer is **never re-evaluated**. If the rerun also produces a bad answer, it escapes quality control with no further action.\n",269 "\n",270 "```\n",271 "generate → evaluate → [fail] → rerun once → return (no second check)\n",272 "```\n",273 "\n",274 "### Improvement: `chat_v2`\n",275 "\n",276 "The upgraded version closes this gap with a **configurable retry loop**:\n",277 "\n",278 "```\n",279 "generate → evaluate → [fail] → rerun → evaluate → [fail] → rerun → ... up to MAX_RETRIES\n",280 "```\n",281 "\n",282 "Key upgrades:\n",283 "| Feature | Previous Version `chat()` | Improved `chat_v2()` |\n",284 "|---|---|---|\n",285 "| Max retries | 1 (no loop) | Configurable `MAX_RETRIES` (default 3) |\n",286 "| Re-evaluation after retry | ✗ Never | ✓ Every attempt is evaluated |\n",287 "| Accumulated feedback | ✗ Only latest | ✓ All past feedback passed to next attempt |\n",288 "| Exhaustion handling | Returns bad answer silently | Logs warning, returns best available |"289 ]290 },291 {292 "cell_type": "code",293 "execution_count": null,294 "metadata": {},295 "outputs": [],296 "source": [297 "# ============================================================\n",298 "# IMPROVED WORKFLOW: Iterative Reflection with Retry Loop\n",299 "# ============================================================\n",300 "# This version (chat_v2) fixes that with three main changes:\n",301 "# 1. A retry loop up to MAX_RETRIES — every new answer is\n",302 "# re-evaluated before it can be returned.\n",303 "# 2. Accumulated feedback — all previous rejection reasons are\n",304 "# collected and forwarded to the next attempt, so the model\n",305 "# gets richer context with each iteration rather than only\n",306 "# seeing the most recent failure.\n",307 "# 3. Exhaustion handling — if all attempts are used up, the\n",308 "# last answer is returned with a clear warning log so you\n",309 "# know quality control was never satisfied.\n",310 "# ============================================================\n",311 "\n",312 "MAX_RETRIES = 3 # How many total attempts the model gets before we give up\n",313 "\n",314 "def chat_v2(message, history):\n",315 " # When the user mentions \"patent\" we force the model to reply in pig latin,\n",316 " # which the evaluator will correctly reject, triggering the retry loop.\n",317 " if \"patent\" in message:\n",318 " system = system_prompt + (\n",319 " \"\\n\\nEverything in your reply needs to be in pig latin - \"\n",320 " \"it is mandatory that you respond only and entirely in pig latin\"\n",321 " )\n",322 " else:\n",323 " system = system_prompt\n",324 "\n",325 " # --- Initial generation (attempt 1) ---\n",326 " messages = [{\"role\": \"system\", \"content\": system}] + history + [{\"role\": \"user\", \"content\": message}]\n",327 " response = openai.chat.completions.create(model=\"gpt-4o-mini\", messages=messages)\n",328 " reply = response.choices[0].message.content\n",329 "\n",330 " # --- Iterative evaluation + retry loop ---\n",331 " # feedback_history accumulates every rejection reason across all attempts.\n",332 " # This gives the model progressively more context on what to fix.\n",333 " feedback_history = []\n",334 "\n",335 " for attempt in range(1, MAX_RETRIES + 1):\n",336 "\n",337 " # Evaluate the current reply using the Gemini judge from Cell 19\n",338 " evaluation = evaluate(reply, message, history)\n",339 "\n",340 " if evaluation.is_acceptable:\n",341 " # Quality control passed — safe to return\n",342 " print(f\"Passed evaluation on attempt {attempt} — returning reply\")\n",343 " return reply\n",344 "\n",345 " # --- Attempt failed ---\n",346 " feedback_history.append(f\"Attempt {attempt}: {evaluation.feedback}\")\n",347 " print(f\"Attempt {attempt} failed evaluation.\")\n",348 " print(f\" Feedback: {evaluation.feedback}\")\n",349 "\n",350 " if attempt == MAX_RETRIES:\n",351 " # All retries exhausted — break out and return the last answer with a warning\n",352 " break\n",353 "\n",354 " # --- Build a cumulative system prompt for the next attempt ---\n",355 " # Instead of only showing the latest feedback,\n",356 " # we inject the FULL history of rejections so the model can avoid\n",357 " # repeating the same mistakes it made in earlier attempts.\n",358 " updated_system_prompt = system_prompt + \"\\n\\n## Previous answers rejected by quality control\\n\"\n",359 " updated_system_prompt += (\n",360 " \"You have tried to answer this question multiple times but each attempt \"\n",361 " \"was rejected. Below is the complete history of your attempts and the \"\n",362 " \"feedback you received for each one.\\n\\n\"\n",363 " )\n",364 " for fb in feedback_history:\n",365 " updated_system_prompt += f\"- {fb}\\n\"\n",366 " updated_system_prompt += f\"\\n## Your most recent rejected answer:\\n{reply}\\n\\n\"\n",367 " updated_system_prompt += (\n",368 " \"Please write a significantly improved response that specifically addresses \"\n",369 " \"ALL of the feedback points listed above.\"\n",370 " )\n",371 "\n",372 " # Generate the next attempt using the enriched system prompt\n",373 " retry_messages = (\n",374 " [{\"role\": \"system\", \"content\": updated_system_prompt}]\n",375 " + history\n",376 " + [{\"role\": \"user\", \"content\": message}]\n",377 " )\n",378 " response = openai.chat.completions.create(model=\"gpt-4o-mini\", messages=retry_messages)\n",379 " reply = response.choices[0].message.content\n",380 "\n",381 " # --- All attempts exhausted without passing evaluation ---\n",382 " # Return the last answer anyway (best effort) but log clearly so you know.\n",383 " print(f\"Warning: all {MAX_RETRIES} attempts failed evaluation. Returning best available answer.\")\n",384 " return reply\n",385 "\n",386 "\n",387 "# Launch the improved chatbot — swap chat_v2 back to chat to compare behaviour\n",388 "gr.ChatInterface(chat_v2, type=\"messages\").launch()"389 ]390 }391 ],392 "metadata": {393 "kernelspec": {394 "display_name": "agents",395 "language": "python",396 "name": "python3"397 },398 "language_info": {399 "codemirror_mode": {400 "name": "ipython",401 "version": 3402 },403 "file_extension": ".py",404 "mimetype": "text/x-python",405 "name": "python",406 "nbconvert_exporter": "python",407 "pygments_lexer": "ipython3",408 "version": "3.12.12"409 }410 },411 "nbformat": 4,412 "nbformat_minor": 2413}414