maxazcona/crewai_docs
012
1[
2 {
3 "id": "b601e3ef-5bcb-4732-aa4d-92dea2aa0314",
4 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
5 "title": "Patronus AI Evaluation - CrewAI",
6 "chunk_index": 0,
7 "text": "CrewAI\n home page\nEnglish\nSearch...\n⌘\nK\nAsk AI\nStart Cloud Trial\ncrewAIInc\n/\ncrewAI\ncrewAIInc\n/\ncrewAI\nSearch...\nNavigation\nObservability\nPatronus AI Evaluation\nDocumentation\nEnterprise\nAPI Reference\nExamples\nWebsite\nForum\nCrew GPT\nGet Help\nReleases\nGet Started\nIntroduction\nInstallation\nQuickstart\nGuides\nStrategy\nAgents\nCrews\nFlows\nAdvanced\nCore Concepts\nAgents\nTasks\nCrews\nFlows\nKnowledge\nLLMs\nProcesses\nCollaboration\nTraining\nMemory\nReasoning\nPlanning\nTesting\nCLI\nTools\nEvent Listeners\nMCP Integration\nMCP Servers as Tools in CrewAI\nStdio Transport\nSSE Transport\nStreamable HTTP Transport\nConnecting to Multiple MCP Servers\nMCP Security Considerations\nTools\nTools Overview\nFile & Document\nWeb Scraping & Browsing\nSearch & Research\nDatabase & Data\nAI & Machine Learning\nCloud & Storage\nAutomation & Integration\nObservability\nOverview\nAgentOps Integration\nArize Phoenix\nLangfuse Integration\nLangtrace Integration\nMaxim Integration\nMLflow Integration\nOpenLIT Integration\nOpik Integration",
8 "metadata": {
9 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
10 "title": "Patronus AI Evaluation - CrewAI"
11 }
12 },
13 {
14 "id": "44fca14a-1837-4e0b-a77a-855cd2b3d5c9",
15 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
16 "title": "Patronus AI Evaluation - CrewAI",
17 "chunk_index": 1,
18 "text": "Automation & Integration\nObservability\nOverview\nAgentOps Integration\nArize Phoenix\nLangfuse Integration\nLangtrace Integration\nMaxim Integration\nMLflow Integration\nOpenLIT Integration\nOpik Integration\nPatronus AI Evaluation\nPortkey Integration\nWeave Integration\nLearn\nOverview\nStrategic LLM Selection Guide\nConditional Tasks\nCoding Agents\nCreate Custom Tools\nCustom LLM Implementation\nCustom Manager Agent\nCustomize Agents\nImage Generation with DALL-E\nForce Tool Output as Result\nHierarchical Process\nHuman Input on Execution\nKickoff Crew Asynchronously\nKickoff Crew for Each\nConnect to any LLM\nUsing Multimodal Agents\nReplay Tasks from Latest Crew Kickoff\nSequential Processes\nUsing Annotations in crew.py\nTelemetry\nTelemetry\nObservability\nPatronus AI Evaluation\nCopy page\nMonitor and evaluate CrewAI agent performance using Patronus AI’s comprehensive evaluation platform for LLM outputs and agent behaviors.\n\nPatronus AI Evaluation",
19 "metadata": {
20 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
21 "title": "Patronus AI Evaluation - CrewAI"
22 }
23 },
24 {
25 "id": "0d30ae60-bd62-48bf-b59d-2293bfd882df",
26 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
27 "title": "Patronus AI Evaluation - CrewAI",
28 "chunk_index": 2,
29 "text": "\nOverview\n\n\nPatronus AI\n provides comprehensive evaluation and monitoring capabilities for CrewAI agents, enabling you to assess model outputs, agent behaviors, and overall system performance. This integration allows you to implement continuous evaluation workflows that help maintain quality and reliability in production environments.\n\n\n\nKey Features\n\n\n\n\nAutomated Evaluation\n: Real-time assessment of agent outputs and behaviors\n\n\nCustom Criteria\n: Define specific evaluation criteria tailored to your use cases\n\n\nPerformance Monitoring\n: Track agent performance metrics over time\n\n\nQuality Assurance\n: Ensure consistent output quality across different scenarios\n\n\nSafety & Compliance\n: Monitor for potential issues and policy violations\n\n\n\n\n\nEvaluation Tools\n\n\nPatronus provides three main evaluation tools for different use cases:\n\n\n\n\nPatronusEvalTool\n: Allows agents to select the most appropriate evaluator and criteria for the evaluation task.",
30 "metadata": {
31 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
32 "title": "Patronus AI Evaluation - CrewAI"
33 }
34 },
35 {
36 "id": "4c7fb670-f367-418d-bd57-f40b34af66da",
37 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
38 "title": "Patronus AI Evaluation - CrewAI",
39 "chunk_index": 3,
40 "text": "Patronus provides three main evaluation tools for different use cases:\n\n\n\n\nPatronusEvalTool\n: Allows agents to select the most appropriate evaluator and criteria for the evaluation task.\n\n\nPatronusPredefinedCriteriaEvalTool\n: Uses predefined evaluator and criteria specified by the user.\n\n\nPatronusLocalEvaluatorTool\n: Uses custom function evaluators defined by the user.\n\n\n\n\n\nInstallation\n\n\nTo use these tools, you need to install the Patronus package:\n\n\nCopy\nAsk AI\nuv\n add\n patronus\n\n\n\n\nYou’ll also need to set up your Patronus API key as an environment variable:\n\n\nCopy\nAsk AI\nexport\n PATRONUS_API_KEY\n=\n\"your_patronus_api_key\"\n\n\n\n\n\nSteps to Get Started\n\n\nTo effectively use the Patronus evaluation tools, follow these steps:\n\n\n\n\nInstall Patronus\n: Install the Patronus package using the command above.\n\n\nSet Up API Key\n: Set your Patronus API key as an environment variable.\n\n\nChoose the Right Tool\n: Select the appropriate Patronus evaluation tool based on your needs.",
41 "metadata": {
42 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
43 "title": "Patronus AI Evaluation - CrewAI"
44 }
45 },
46 {
47 "id": "3277642b-aa81-4012-adc7-2f2581bbaf0f",
48 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
49 "title": "Patronus AI Evaluation - CrewAI",
50 "chunk_index": 4,
51 "text": "Set Up API Key\n: Set your Patronus API key as an environment variable.\n\n\nChoose the Right Tool\n: Select the appropriate Patronus evaluation tool based on your needs.\n\n\nConfigure the Tool\n: Configure the tool with the necessary parameters.\n\n\n\n\n\nExamples\n\n\n\nUsing PatronusEvalTool\n\n\nThe following example demonstrates how to use the \nPatronusEvalTool\n, which allows agents to select the most appropriate evaluator and criteria:\n\n\nCode\nCopy\nAsk AI\nfrom\n crewai \nimport\n Agent, Task, Crew\n\n\nfrom\n crewai_tools \nimport\n PatronusEvalTool\n\n\n\n\n# Initialize the tool\n\n\npatronus_eval_tool \n=\n PatronusEvalTool()\n\n\n\n\n# Define an agent that uses the tool\n\n\ncoding_agent \n=\n Agent(\n\n\n role\n=\n\"Coding Agent\"\n,\n\n\n goal\n=\n\"Generate high quality code and verify that the output is code\"\n,\n\n\n backstory\n=\n\"An experienced coder who can generate high quality python code.\"\n,\n\n\n tools\n=\n[patronus_eval_tool],\n\n\n verbose\n=\nTrue\n,\n\n\n)\n\n\n\n\n# Example task to generate and evaluate code",
52 "metadata": {
53 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
54 "title": "Patronus AI Evaluation - CrewAI"
55 }
56 },
57 {
58 "id": "94ae3c1c-a85b-4b50-b48e-29f723a65c38",
59 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
60 "title": "Patronus AI Evaluation - CrewAI",
61 "chunk_index": 5,
62 "text": "backstory\n=\n\"An experienced coder who can generate high quality python code.\"\n,\n\n\n tools\n=\n[patronus_eval_tool],\n\n\n verbose\n=\nTrue\n,\n\n\n)\n\n\n\n\n# Example task to generate and evaluate code\n\n\ngenerate_code_task \n=\n Task(\n\n\n description\n=\n\"Create a simple program to generate the first N numbers in the Fibonacci sequence. Select the most appropriate evaluator and criteria for evaluating your output.\"\n,\n\n\n expected_output\n=\n\"Program that generates the first N numbers in the Fibonacci sequence.\"\n,\n\n\n agent\n=\ncoding_agent,\n\n\n)\n\n\n\n\n# Create and run the crew\n\n\ncrew \n=\n Crew(\nagents\n=\n[coding_agent], \ntasks\n=\n[generate_code_task])\n\n\nresult \n=\n crew.kickoff()\n\n\n\n\n\nUsing PatronusPredefinedCriteriaEvalTool\n\n\nThe following example demonstrates how to use the \nPatronusPredefinedCriteriaEvalTool\n, which uses predefined evaluator and criteria:\n\n\nCode\nCopy\nAsk AI\nfrom\n crewai \nimport\n Agent, Task, Crew\n\n\nfrom\n crewai_tools \nimport\n PatronusPredefinedCriteriaEvalTool",
63 "metadata": {
64 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
65 "title": "Patronus AI Evaluation - CrewAI"
66 }
67 },
68 {
69 "id": "1599666b-88d1-4912-bfdf-1a782a541045",
70 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
71 "title": "Patronus AI Evaluation - CrewAI",
72 "chunk_index": 6,
73 "text": "Code\nCopy\nAsk AI\nfrom\n crewai \nimport\n Agent, Task, Crew\n\n\nfrom\n crewai_tools \nimport\n PatronusPredefinedCriteriaEvalTool\n\n\n\n\n# Initialize the tool with predefined criteria\n\n\npatronus_eval_tool \n=\n PatronusPredefinedCriteriaEvalTool(\n\n\n evaluators\n=\n[{\n\"evaluator\"\n: \n\"judge\"\n, \n\"criteria\"\n: \n\"contains-code\"\n}]\n\n\n)\n\n\n\n\n# Define an agent that uses the tool\n\n\ncoding_agent \n=\n Agent(\n\n\n role\n=\n\"Coding Agent\"\n,\n\n\n goal\n=\n\"Generate high quality code\"\n,\n\n\n backstory\n=\n\"An experienced coder who can generate high quality python code.\"\n,\n\n\n tools\n=\n[patronus_eval_tool],\n\n\n verbose\n=\nTrue\n,\n\n\n)\n\n\n\n\n# Example task to generate code\n\n\ngenerate_code_task \n=\n Task(\n\n\n description\n=\n\"Create a simple program to generate the first N numbers in the Fibonacci sequence.\"\n,\n\n\n expected_output\n=\n\"Program that generates the first N numbers in the Fibonacci sequence.\"\n,\n\n\n agent\n=\ncoding_agent,\n\n\n)\n\n\n\n\n# Create and run the crew",
74 "metadata": {
75 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
76 "title": "Patronus AI Evaluation - CrewAI"
77 }
78 },
79 {
80 "id": "b904179a-eb0a-4244-b2ff-3d313d6a9a8b",
81 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
82 "title": "Patronus AI Evaluation - CrewAI",
83 "chunk_index": 7,
84 "text": "expected_output\n=\n\"Program that generates the first N numbers in the Fibonacci sequence.\"\n,\n\n\n agent\n=\ncoding_agent,\n\n\n)\n\n\n\n\n# Create and run the crew\n\n\ncrew \n=\n Crew(\nagents\n=\n[coding_agent], \ntasks\n=\n[generate_code_task])\n\n\nresult \n=\n crew.kickoff()\n\n\n\n\n\nUsing PatronusLocalEvaluatorTool\n\n\nThe following example demonstrates how to use the \nPatronusLocalEvaluatorTool\n, which uses custom function evaluators:\n\n\nCode\nCopy\nAsk AI\nfrom\n crewai \nimport\n Agent, Task, Crew\n\n\nfrom\n crewai_tools \nimport\n PatronusLocalEvaluatorTool\n\n\nfrom\n patronus \nimport\n Client, EvaluationResult\n\n\nimport\n random\n\n\n\n\n# Initialize the Patronus client\n\n\nclient \n=\n Client()\n\n\n\n\n# Register a custom evaluator\n\n\n@client.register_local_evaluator\n(\n\"random_evaluator\"\n)\n\n\ndef\n random_evaluator\n(\n**\nkwargs\n):\n\n\n score \n=\n random.random()\n\n\n return\n EvaluationResult(\n\n\n score_raw\n=\nscore,\n\n\n pass_\n=\nscore \n>=\n 0.5\n,\n\n\n explanation\n=\n\"example explanation\"\n,\n\n\n )",
85 "metadata": {
86 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
87 "title": "Patronus AI Evaluation - CrewAI"
88 }
89 },
90 {
91 "id": "6b503dfb-24ac-496e-bc32-969e5a7d5457",
92 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
93 "title": "Patronus AI Evaluation - CrewAI",
94 "chunk_index": 8,
95 "text": "score \n=\n random.random()\n\n\n return\n EvaluationResult(\n\n\n score_raw\n=\nscore,\n\n\n pass_\n=\nscore \n>=\n 0.5\n,\n\n\n explanation\n=\n\"example explanation\"\n,\n\n\n )\n\n\n\n\n# Initialize the tool with the custom evaluator\n\n\npatronus_eval_tool \n=\n PatronusLocalEvaluatorTool(\n\n\n patronus_client\n=\nclient,\n\n\n evaluator\n=\n\"random_evaluator\"\n,\n\n\n evaluated_model_gold_answer\n=\n\"example label\"\n,\n\n\n)\n\n\n\n\n# Define an agent that uses the tool\n\n\ncoding_agent \n=\n Agent(\n\n\n role\n=\n\"Coding Agent\"\n,\n\n\n goal\n=\n\"Generate high quality code\"\n,\n\n\n backstory\n=\n\"An experienced coder who can generate high quality python code.\"\n,\n\n\n tools\n=\n[patronus_eval_tool],\n\n\n verbose\n=\nTrue\n,\n\n\n)\n\n\n\n\n# Example task to generate code\n\n\ngenerate_code_task \n=\n Task(\n\n\n description\n=\n\"Create a simple program to generate the first N numbers in the Fibonacci sequence.\"\n,\n\n\n expected_output\n=\n\"Program that generates the first N numbers in the Fibonacci sequence.\"\n,",
96 "metadata": {
97 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
98 "title": "Patronus AI Evaluation - CrewAI"
99 }
100 },
101 {
102 "id": "2fbbcc5a-6ee1-4198-bfb2-ded96b7c1874",
103 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
104 "title": "Patronus AI Evaluation - CrewAI",
105 "chunk_index": 9,
106 "text": "expected_output\n=\n\"Program that generates the first N numbers in the Fibonacci sequence.\"\n,\n\n\n agent\n=\ncoding_agent,\n\n\n)\n\n\n\n\n# Create and run the crew\n\n\ncrew \n=\n Crew(\nagents\n=\n[coding_agent], \ntasks\n=\n[generate_code_task])\n\n\nresult \n=\n crew.kickoff()\n\n\n\n\n\nParameters\n\n\n\nPatronusEvalTool\n\n\nThe \nPatronusEvalTool\n does not require any parameters during initialization. It automatically fetches available evaluators and criteria from the Patronus API.\n\n\n\nPatronusPredefinedCriteriaEvalTool\n\n\nThe \nPatronusPredefinedCriteriaEvalTool\n accepts the following parameters during initialization:\n\n\n\n\nevaluators\n: Required. A list of dictionaries containing the evaluator and criteria to use. For example: \n[{\"evaluator\": \"judge\", \"criteria\": \"contains-code\"}]\n.\n\n\n\n\n\nPatronusLocalEvaluatorTool\n\n\nThe \nPatronusLocalEvaluatorTool\n accepts the following parameters during initialization:\n\n\n\n\npatronus_client\n: Required. The Patronus client instance.",
107 "metadata": {
108 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
109 "title": "Patronus AI Evaluation - CrewAI"
110 }
111 },
112 {
113 "id": "a4c4dfd3-80d2-4471-af60-0007f7e8ed7f",
114 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
115 "title": "Patronus AI Evaluation - CrewAI",
116 "chunk_index": 10,
117 "text": "\nPatronusLocalEvaluatorTool\n\n\nThe \nPatronusLocalEvaluatorTool\n accepts the following parameters during initialization:\n\n\n\n\npatronus_client\n: Required. The Patronus client instance.\n\n\nevaluator\n: Optional. The name of the registered local evaluator to use. Default is an empty string.\n\n\nevaluated_model_gold_answer\n: Optional. The gold answer to use for evaluation. Default is an empty string.\n\n\n\n\n\nUsage\n\n\nWhen using the Patronus evaluation tools, you provide the model input, output, and context, and the tool returns the evaluation results from the Patronus API.\n\n\nFor the \nPatronusEvalTool\n and \nPatronusPredefinedCriteriaEvalTool\n, the following parameters are required when calling the tool:\n\n\n\n\nevaluated_model_input\n: The agent’s task description in simple text.\n\n\nevaluated_model_output\n: The agent’s output of the task.\n\n\nevaluated_model_retrieved_context\n: The agent’s context.",
118 "metadata": {
119 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
120 "title": "Patronus AI Evaluation - CrewAI"
121 }
122 },
123 {
124 "id": "33c6c697-801d-4538-a651-cebf27e63f89",
125 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
126 "title": "Patronus AI Evaluation - CrewAI",
127 "chunk_index": 11,
128 "text": "evaluated_model_input\n: The agent’s task description in simple text.\n\n\nevaluated_model_output\n: The agent’s output of the task.\n\n\nevaluated_model_retrieved_context\n: The agent’s context.\n\n\n\n\nFor the \nPatronusLocalEvaluatorTool\n, the same parameters are required, but the evaluator and gold answer are specified during initialization.\n\n\n\nConclusion",
129 "metadata": {
130 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
131 "title": "Patronus AI Evaluation - CrewAI"
132 }
133 },
134 {
135 "id": "c2fc4dd4-9aa9-455c-9f80-a7ff344ab614",
136 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
137 "title": "Patronus AI Evaluation - CrewAI",
138 "chunk_index": 12,
139 "text": "The Patronus evaluation tools provide a powerful way to evaluate and score model inputs and outputs using the Patronus AI platform. By enabling agents to evaluate their own outputs or the outputs of other agents, these tools can help improve the quality and reliability of CrewAI workflows.\nAssistant\nResponses are generated using AI and may contain mistakes.\nWas this page helpful?\nYes\nNo\nOpik Integration\nPortkey Integration\nwebsite\nx\ngithub\nlinkedin\nyoutube\nreddit\nPowered by Mintlify\nOn this page\nPatronus AI Evaluation\nOverview\nKey Features\nEvaluation Tools\nInstallation\nSteps to Get Started\nExamples\nUsing PatronusEvalTool\nUsing PatronusPredefinedCriteriaEvalTool\nUsing PatronusLocalEvaluatorTool\nParameters\nPatronusEvalTool\nPatronusPredefinedCriteriaEvalTool\nPatronusLocalEvaluatorTool\nUsage\nConclusion\nObservability\nPatronus AI Evaluation\nCopy page\nMonitor and evaluate CrewAI agent performance using Patronus AI’s comprehensive evaluation platform for LLM outputs and agent behaviors.\n",
140 "metadata": {
141 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
142 "title": "Patronus AI Evaluation - CrewAI"
143 }
144 },
145 {
146 "id": "3fda3c89-e29b-4c37-9dd8-d1ae4f687108",
147 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
148 "title": "Patronus AI Evaluation - CrewAI",
149 "chunk_index": 13,
150 "text": "Conclusion\nObservability\nPatronus AI Evaluation\nCopy page\nMonitor and evaluate CrewAI agent performance using Patronus AI’s comprehensive evaluation platform for LLM outputs and agent behaviors.\n\nPatronus AI Evaluation",
151 "metadata": {
152 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
153 "title": "Patronus AI Evaluation - CrewAI"
154 }
155 },
156 {
157 "id": "1acb4f80-cdb8-4468-98f2-465d6613b24d",
158 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
159 "title": "Patronus AI Evaluation - CrewAI",
160 "chunk_index": 14,
161 "text": "\nOverview\n\n\nPatronus AI\n provides comprehensive evaluation and monitoring capabilities for CrewAI agents, enabling you to assess model outputs, agent behaviors, and overall system performance. This integration allows you to implement continuous evaluation workflows that help maintain quality and reliability in production environments.\n\n\n\nKey Features\n\n\n\n\nAutomated Evaluation\n: Real-time assessment of agent outputs and behaviors\n\n\nCustom Criteria\n: Define specific evaluation criteria tailored to your use cases\n\n\nPerformance Monitoring\n: Track agent performance metrics over time\n\n\nQuality Assurance\n: Ensure consistent output quality across different scenarios\n\n\nSafety & Compliance\n: Monitor for potential issues and policy violations\n\n\n\n\n\nEvaluation Tools\n\n\nPatronus provides three main evaluation tools for different use cases:\n\n\n\n\nPatronusEvalTool\n: Allows agents to select the most appropriate evaluator and criteria for the evaluation task.",
162 "metadata": {
163 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
164 "title": "Patronus AI Evaluation - CrewAI"
165 }
166 },
167 {
168 "id": "11a6ab52-5f44-48f3-96a5-d4f2b09dc60f",
169 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
170 "title": "Patronus AI Evaluation - CrewAI",
171 "chunk_index": 15,
172 "text": "Patronus provides three main evaluation tools for different use cases:\n\n\n\n\nPatronusEvalTool\n: Allows agents to select the most appropriate evaluator and criteria for the evaluation task.\n\n\nPatronusPredefinedCriteriaEvalTool\n: Uses predefined evaluator and criteria specified by the user.\n\n\nPatronusLocalEvaluatorTool\n: Uses custom function evaluators defined by the user.\n\n\n\n\n\nInstallation\n\n\nTo use these tools, you need to install the Patronus package:\n\n\nCopy\nAsk AI\nuv\n add\n patronus\n\n\n\n\nYou’ll also need to set up your Patronus API key as an environment variable:\n\n\nCopy\nAsk AI\nexport\n PATRONUS_API_KEY\n=\n\"your_patronus_api_key\"\n\n\n\n\n\nSteps to Get Started\n\n\nTo effectively use the Patronus evaluation tools, follow these steps:\n\n\n\n\nInstall Patronus\n: Install the Patronus package using the command above.\n\n\nSet Up API Key\n: Set your Patronus API key as an environment variable.\n\n\nChoose the Right Tool\n: Select the appropriate Patronus evaluation tool based on your needs.",
173 "metadata": {
174 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
175 "title": "Patronus AI Evaluation - CrewAI"
176 }
177 },
178 {
179 "id": "658365a4-b845-4bae-9409-64b3c931da85",
180 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
181 "title": "Patronus AI Evaluation - CrewAI",
182 "chunk_index": 16,
183 "text": "Set Up API Key\n: Set your Patronus API key as an environment variable.\n\n\nChoose the Right Tool\n: Select the appropriate Patronus evaluation tool based on your needs.\n\n\nConfigure the Tool\n: Configure the tool with the necessary parameters.\n\n\n\n\n\nExamples\n\n\n\nUsing PatronusEvalTool\n\n\nThe following example demonstrates how to use the \nPatronusEvalTool\n, which allows agents to select the most appropriate evaluator and criteria:\n\n\nCode\nCopy\nAsk AI\nfrom\n crewai \nimport\n Agent, Task, Crew\n\n\nfrom\n crewai_tools \nimport\n PatronusEvalTool\n\n\n\n\n# Initialize the tool\n\n\npatronus_eval_tool \n=\n PatronusEvalTool()\n\n\n\n\n# Define an agent that uses the tool\n\n\ncoding_agent \n=\n Agent(\n\n\n role\n=\n\"Coding Agent\"\n,\n\n\n goal\n=\n\"Generate high quality code and verify that the output is code\"\n,\n\n\n backstory\n=\n\"An experienced coder who can generate high quality python code.\"\n,\n\n\n tools\n=\n[patronus_eval_tool],\n\n\n verbose\n=\nTrue\n,\n\n\n)\n\n\n\n\n# Example task to generate and evaluate code",
184 "metadata": {
185 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
186 "title": "Patronus AI Evaluation - CrewAI"
187 }
188 },
189 {
190 "id": "7f1d82ac-cba7-48d3-ab2b-96406fb5b336",
191 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
192 "title": "Patronus AI Evaluation - CrewAI",
193 "chunk_index": 17,
194 "text": "backstory\n=\n\"An experienced coder who can generate high quality python code.\"\n,\n\n\n tools\n=\n[patronus_eval_tool],\n\n\n verbose\n=\nTrue\n,\n\n\n)\n\n\n\n\n# Example task to generate and evaluate code\n\n\ngenerate_code_task \n=\n Task(\n\n\n description\n=\n\"Create a simple program to generate the first N numbers in the Fibonacci sequence. Select the most appropriate evaluator and criteria for evaluating your output.\"\n,\n\n\n expected_output\n=\n\"Program that generates the first N numbers in the Fibonacci sequence.\"\n,\n\n\n agent\n=\ncoding_agent,\n\n\n)\n\n\n\n\n# Create and run the crew\n\n\ncrew \n=\n Crew(\nagents\n=\n[coding_agent], \ntasks\n=\n[generate_code_task])\n\n\nresult \n=\n crew.kickoff()\n\n\n\n\n\nUsing PatronusPredefinedCriteriaEvalTool\n\n\nThe following example demonstrates how to use the \nPatronusPredefinedCriteriaEvalTool\n, which uses predefined evaluator and criteria:\n\n\nCode\nCopy\nAsk AI\nfrom\n crewai \nimport\n Agent, Task, Crew\n\n\nfrom\n crewai_tools \nimport\n PatronusPredefinedCriteriaEvalTool",
195 "metadata": {
196 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
197 "title": "Patronus AI Evaluation - CrewAI"
198 }
199 },
200 {
201 "id": "0c56e7ef-80ae-4f84-bb6c-692ae24a1a2f",
202 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
203 "title": "Patronus AI Evaluation - CrewAI",
204 "chunk_index": 18,
205 "text": "Code\nCopy\nAsk AI\nfrom\n crewai \nimport\n Agent, Task, Crew\n\n\nfrom\n crewai_tools \nimport\n PatronusPredefinedCriteriaEvalTool\n\n\n\n\n# Initialize the tool with predefined criteria\n\n\npatronus_eval_tool \n=\n PatronusPredefinedCriteriaEvalTool(\n\n\n evaluators\n=\n[{\n\"evaluator\"\n: \n\"judge\"\n, \n\"criteria\"\n: \n\"contains-code\"\n}]\n\n\n)\n\n\n\n\n# Define an agent that uses the tool\n\n\ncoding_agent \n=\n Agent(\n\n\n role\n=\n\"Coding Agent\"\n,\n\n\n goal\n=\n\"Generate high quality code\"\n,\n\n\n backstory\n=\n\"An experienced coder who can generate high quality python code.\"\n,\n\n\n tools\n=\n[patronus_eval_tool],\n\n\n verbose\n=\nTrue\n,\n\n\n)\n\n\n\n\n# Example task to generate code\n\n\ngenerate_code_task \n=\n Task(\n\n\n description\n=\n\"Create a simple program to generate the first N numbers in the Fibonacci sequence.\"\n,\n\n\n expected_output\n=\n\"Program that generates the first N numbers in the Fibonacci sequence.\"\n,\n\n\n agent\n=\ncoding_agent,\n\n\n)\n\n\n\n\n# Create and run the crew",
206 "metadata": {
207 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
208 "title": "Patronus AI Evaluation - CrewAI"
209 }
210 },
211 {
212 "id": "2d4d673a-407e-480a-96e1-d2b7cd4160f3",
213 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
214 "title": "Patronus AI Evaluation - CrewAI",
215 "chunk_index": 19,
216 "text": "expected_output\n=\n\"Program that generates the first N numbers in the Fibonacci sequence.\"\n,\n\n\n agent\n=\ncoding_agent,\n\n\n)\n\n\n\n\n# Create and run the crew\n\n\ncrew \n=\n Crew(\nagents\n=\n[coding_agent], \ntasks\n=\n[generate_code_task])\n\n\nresult \n=\n crew.kickoff()\n\n\n\n\n\nUsing PatronusLocalEvaluatorTool\n\n\nThe following example demonstrates how to use the \nPatronusLocalEvaluatorTool\n, which uses custom function evaluators:\n\n\nCode\nCopy\nAsk AI\nfrom\n crewai \nimport\n Agent, Task, Crew\n\n\nfrom\n crewai_tools \nimport\n PatronusLocalEvaluatorTool\n\n\nfrom\n patronus \nimport\n Client, EvaluationResult\n\n\nimport\n random\n\n\n\n\n# Initialize the Patronus client\n\n\nclient \n=\n Client()\n\n\n\n\n# Register a custom evaluator\n\n\n@client.register_local_evaluator\n(\n\"random_evaluator\"\n)\n\n\ndef\n random_evaluator\n(\n**\nkwargs\n):\n\n\n score \n=\n random.random()\n\n\n return\n EvaluationResult(\n\n\n score_raw\n=\nscore,\n\n\n pass_\n=\nscore \n>=\n 0.5\n,\n\n\n explanation\n=\n\"example explanation\"\n,\n\n\n )",
217 "metadata": {
218 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
219 "title": "Patronus AI Evaluation - CrewAI"
220 }
221 },
222 {
223 "id": "4b61c369-292f-465f-b513-b2e275c76497",
224 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
225 "title": "Patronus AI Evaluation - CrewAI",
226 "chunk_index": 20,
227 "text": "score \n=\n random.random()\n\n\n return\n EvaluationResult(\n\n\n score_raw\n=\nscore,\n\n\n pass_\n=\nscore \n>=\n 0.5\n,\n\n\n explanation\n=\n\"example explanation\"\n,\n\n\n )\n\n\n\n\n# Initialize the tool with the custom evaluator\n\n\npatronus_eval_tool \n=\n PatronusLocalEvaluatorTool(\n\n\n patronus_client\n=\nclient,\n\n\n evaluator\n=\n\"random_evaluator\"\n,\n\n\n evaluated_model_gold_answer\n=\n\"example label\"\n,\n\n\n)\n\n\n\n\n# Define an agent that uses the tool\n\n\ncoding_agent \n=\n Agent(\n\n\n role\n=\n\"Coding Agent\"\n,\n\n\n goal\n=\n\"Generate high quality code\"\n,\n\n\n backstory\n=\n\"An experienced coder who can generate high quality python code.\"\n,\n\n\n tools\n=\n[patronus_eval_tool],\n\n\n verbose\n=\nTrue\n,\n\n\n)\n\n\n\n\n# Example task to generate code\n\n\ngenerate_code_task \n=\n Task(\n\n\n description\n=\n\"Create a simple program to generate the first N numbers in the Fibonacci sequence.\"\n,\n\n\n expected_output\n=\n\"Program that generates the first N numbers in the Fibonacci sequence.\"\n,",
228 "metadata": {
229 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
230 "title": "Patronus AI Evaluation - CrewAI"
231 }
232 },
233 {
234 "id": "2243df49-cac2-40c8-b70e-507d21f68261",
235 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
236 "title": "Patronus AI Evaluation - CrewAI",
237 "chunk_index": 21,
238 "text": "expected_output\n=\n\"Program that generates the first N numbers in the Fibonacci sequence.\"\n,\n\n\n agent\n=\ncoding_agent,\n\n\n)\n\n\n\n\n# Create and run the crew\n\n\ncrew \n=\n Crew(\nagents\n=\n[coding_agent], \ntasks\n=\n[generate_code_task])\n\n\nresult \n=\n crew.kickoff()\n\n\n\n\n\nParameters\n\n\n\nPatronusEvalTool\n\n\nThe \nPatronusEvalTool\n does not require any parameters during initialization. It automatically fetches available evaluators and criteria from the Patronus API.\n\n\n\nPatronusPredefinedCriteriaEvalTool\n\n\nThe \nPatronusPredefinedCriteriaEvalTool\n accepts the following parameters during initialization:\n\n\n\n\nevaluators\n: Required. A list of dictionaries containing the evaluator and criteria to use. For example: \n[{\"evaluator\": \"judge\", \"criteria\": \"contains-code\"}]\n.\n\n\n\n\n\nPatronusLocalEvaluatorTool\n\n\nThe \nPatronusLocalEvaluatorTool\n accepts the following parameters during initialization:\n\n\n\n\npatronus_client\n: Required. The Patronus client instance.",
239 "metadata": {
240 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
241 "title": "Patronus AI Evaluation - CrewAI"
242 }
243 },
244 {
245 "id": "a94b2859-1b7e-4a5e-a0f5-d9c329317cbc",
246 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
247 "title": "Patronus AI Evaluation - CrewAI",
248 "chunk_index": 22,
249 "text": "\nPatronusLocalEvaluatorTool\n\n\nThe \nPatronusLocalEvaluatorTool\n accepts the following parameters during initialization:\n\n\n\n\npatronus_client\n: Required. The Patronus client instance.\n\n\nevaluator\n: Optional. The name of the registered local evaluator to use. Default is an empty string.\n\n\nevaluated_model_gold_answer\n: Optional. The gold answer to use for evaluation. Default is an empty string.\n\n\n\n\n\nUsage\n\n\nWhen using the Patronus evaluation tools, you provide the model input, output, and context, and the tool returns the evaluation results from the Patronus API.\n\n\nFor the \nPatronusEvalTool\n and \nPatronusPredefinedCriteriaEvalTool\n, the following parameters are required when calling the tool:\n\n\n\n\nevaluated_model_input\n: The agent’s task description in simple text.\n\n\nevaluated_model_output\n: The agent’s output of the task.\n\n\nevaluated_model_retrieved_context\n: The agent’s context.",
250 "metadata": {
251 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
252 "title": "Patronus AI Evaluation - CrewAI"
253 }
254 },
255 {
256 "id": "d5e9ee94-8ab2-419f-96ac-60b671a6aebe",
257 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
258 "title": "Patronus AI Evaluation - CrewAI",
259 "chunk_index": 23,
260 "text": "evaluated_model_input\n: The agent’s task description in simple text.\n\n\nevaluated_model_output\n: The agent’s output of the task.\n\n\nevaluated_model_retrieved_context\n: The agent’s context.\n\n\n\n\nFor the \nPatronusLocalEvaluatorTool\n, the same parameters are required, but the evaluator and gold answer are specified during initialization.\n\n\n\nConclusion",
261 "metadata": {
262 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
263 "title": "Patronus AI Evaluation - CrewAI"
264 }
265 },
266 {
267 "id": "1753853f-0219-4a09-85b4-7b081d39afd3",
268 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
269 "title": "Patronus AI Evaluation - CrewAI",
270 "chunk_index": 24,
271 "text": "For the \nPatronusLocalEvaluatorTool\n, the same parameters are required, but the evaluator and gold answer are specified during initialization.\n\n\n\nConclusion\n\n\nThe Patronus evaluation tools provide a powerful way to evaluate and score model inputs and outputs using the Patronus AI platform. By enabling agents to evaluate their own outputs or the outputs of other agents, these tools can help improve the quality and reliability of CrewAI workflows.\nAssistant\nResponses are generated using AI and may contain mistakes.\nWas this page helpful?\nYes\nNo\nOpik Integration\nPortkey Integration\nwebsite\nx\ngithub\nlinkedin\nyoutube\nreddit\nPowered by Mintlify\nOn this page\nPatronus AI Evaluation\nOverview\nKey Features\nEvaluation Tools\nInstallation\nSteps to Get Started\nExamples\nUsing PatronusEvalTool\nUsing PatronusPredefinedCriteriaEvalTool\nUsing PatronusLocalEvaluatorTool\nParameters\nPatronusEvalTool\nPatronusPredefinedCriteriaEvalTool\nPatronusLocalEvaluatorTool\nUsage\nConclusion",
272 "metadata": {
273 "source": "https://docs.crewai.com/en/observability/patronus-evaluation",
274 "title": "Patronus AI Evaluation - CrewAI"
275 }
276 },
277 {
278 "id": "799d6b3d-a5b7-4aa0-a8da-0218bfc1074f",
279 "source": "https://docs.crewai.com/en/concepts/reasoning",
280 "title": "Reasoning - CrewAI",
281 "chunk_index": 0,
282 "text": "CrewAI\n home page\nEnglish\nSearch...\n⌘\nK\nAsk AI\nStart Cloud Trial\ncrewAIInc\n/\ncrewAI\ncrewAIInc\n/\ncrewAI\nSearch...\nNavigation\nCore Concepts\nReasoning\nDocumentation\nEnterprise\nAPI Reference\nExamples\nWebsite\nForum\nCrew GPT\nGet Help\nReleases\nGet Started\nIntroduction\nInstallation\nQuickstart\nGuides\nStrategy\nAgents\nCrews\nFlows\nAdvanced\nCore Concepts\nAgents\nTasks\nCrews\nFlows\nKnowledge\nLLMs\nProcesses\nCollaboration\nTraining\nMemory\nReasoning\nPlanning\nTesting\nCLI\nTools\nEvent Listeners\nMCP Integration\nMCP Servers as Tools in CrewAI\nStdio Transport\nSSE Transport\nStreamable HTTP Transport\nConnecting to Multiple MCP Servers\nMCP Security Considerations\nTools\nTools Overview\nFile & Document\nWeb Scraping & Browsing\nSearch & Research\nDatabase & Data\nAI & Machine Learning\nCloud & Storage\nAutomation & Integration\nObservability\nOverview\nAgentOps Integration\nArize Phoenix\nLangfuse Integration\nLangtrace Integration\nMaxim Integration\nMLflow Integration\nOpenLIT Integration\nOpik Integration\nPatronus AI Evaluation",
283 "metadata": {
284 "source": "https://docs.crewai.com/en/concepts/reasoning",
285 "title": "Reasoning - CrewAI"
286 }
287 },
288 {
289 "id": "c6addbc3-1fb7-4e46-aa5b-a88b73bea787",
290 "source": "https://docs.crewai.com/en/concepts/reasoning",
291 "title": "Reasoning - CrewAI",
292 "chunk_index": 1,
293 "text": "Observability\nOverview\nAgentOps Integration\nArize Phoenix\nLangfuse Integration\nLangtrace Integration\nMaxim Integration\nMLflow Integration\nOpenLIT Integration\nOpik Integration\nPatronus AI Evaluation\nPortkey Integration\nWeave Integration\nLearn\nOverview\nStrategic LLM Selection Guide\nConditional Tasks\nCoding Agents\nCreate Custom Tools\nCustom LLM Implementation\nCustom Manager Agent\nCustomize Agents\nImage Generation with DALL-E\nForce Tool Output as Result\nHierarchical Process\nHuman Input on Execution\nKickoff Crew Asynchronously\nKickoff Crew for Each\nConnect to any LLM\nUsing Multimodal Agents\nReplay Tasks from Latest Crew Kickoff\nSequential Processes\nUsing Annotations in crew.py\nTelemetry\nTelemetry\nCore Concepts\nReasoning\nCopy page\nLearn how to enable and use agent reasoning to improve task execution.\n\nOverview",
294 "metadata": {
295 "source": "https://docs.crewai.com/en/concepts/reasoning",
296 "title": "Reasoning - CrewAI"
297 }
298 },
299 {
300 "id": "689a986b-d82c-44f4-bacc-ac4fc0ee53e1",
301 "source": "https://docs.crewai.com/en/concepts/reasoning",
302 "title": "Reasoning - CrewAI",
303 "chunk_index": 2,
304 "text": "Agent reasoning is a feature that allows agents to reflect on a task and create a plan before execution. This helps agents approach tasks more methodically and ensures they’re ready to perform the assigned work.\n\n\n\nUsage\n\n\nTo enable reasoning for an agent, simply set \nreasoning=True\n when creating the agent:\n\n\nCopy\nAsk AI\nfrom\n crewai \nimport\n Agent\n\n\n\n\nagent \n=\n Agent(\n\n\n role\n=\n\"Data Analyst\"\n,\n\n\n goal\n=\n\"Analyze complex datasets and provide insights\"\n,\n\n\n backstory\n=\n\"You are an experienced data analyst with expertise in finding patterns in complex data.\"\n,\n\n\n reasoning\n=\nTrue\n, \n# Enable reasoning\n\n\n max_reasoning_attempts\n=\n3\n # Optional: Set a maximum number of reasoning attempts\n\n\n)\n\n\n\n\n\nHow It Works\n\n\nWhen reasoning is enabled, before executing a task, the agent will:\n\n\n\n\nReflect on the task and create a detailed plan\n\n\nEvaluate whether it’s ready to execute the task\n\n\nRefine the plan as necessary until it’s ready or max_reasoning_attempts is reached",
305 "metadata": {
306 "source": "https://docs.crewai.com/en/concepts/reasoning",
307 "title": "Reasoning - CrewAI"
308 }
309 },
310 {
311 "id": "7f3260ed-724a-4c74-8d67-42c263fa50c5",
312 "source": "https://docs.crewai.com/en/concepts/reasoning",
313 "title": "Reasoning - CrewAI",
314 "chunk_index": 3,
315 "text": "Reflect on the task and create a detailed plan\n\n\nEvaluate whether it’s ready to execute the task\n\n\nRefine the plan as necessary until it’s ready or max_reasoning_attempts is reached\n\n\nInject the reasoning plan into the task description before execution\n\n\n\n\nThis process helps the agent break down complex tasks into manageable steps and identify potential challenges before starting.\n\n\n\nConfiguration Options\n\n\n\nreasoning\nbool\ndefault:\n\"False\"\nEnable or disable reasoning\n\n\n\nmax_reasoning_attempts\nint\ndefault:\n\"None\"\nMaximum number of attempts to refine the plan before proceeding with execution. If None (default), the agent will continue refining until it’s ready.\n\n\n\nExample\n\n\nHere’s a complete example:\n\n\nCopy\nAsk AI\nfrom\n crewai \nimport\n Agent, Task, Crew\n\n\n\n\n# Create an agent with reasoning enabled\n\n\nanalyst \n=\n Agent(\n\n\n role\n=\n\"Data Analyst\"\n,\n\n\n goal\n=\n\"Analyze data and provide insights\"\n,\n\n\n backstory\n=\n\"You are an expert data analyst.\"\n,\n\n\n reasoning\n=\nTrue\n,",
316 "metadata": {
317 "source": "https://docs.crewai.com/en/concepts/reasoning",
318 "title": "Reasoning - CrewAI"
319 }
320 },
321 {
322 "id": "b900ef35-3f7f-485e-b5a9-783475869f1a",
323 "source": "https://docs.crewai.com/en/concepts/reasoning",
324 "title": "Reasoning - CrewAI",
325 "chunk_index": 4,
326 "text": "analyst \n=\n Agent(\n\n\n role\n=\n\"Data Analyst\"\n,\n\n\n goal\n=\n\"Analyze data and provide insights\"\n,\n\n\n backstory\n=\n\"You are an expert data analyst.\"\n,\n\n\n reasoning\n=\nTrue\n,\n\n\n max_reasoning_attempts\n=\n3\n # Optional: Set a limit on reasoning attempts\n\n\n)\n\n\n\n\n# Create a task\n\n\nanalysis_task \n=\n Task(\n\n\n description\n=\n\"Analyze the provided sales data and identify key trends.\"\n,\n\n\n expected_output\n=\n\"A report highlighting the top 3 sales trends.\"\n,\n\n\n agent\n=\nanalyst\n\n\n)\n\n\n\n\n# Create a crew and run the task\n\n\ncrew \n=\n Crew(\nagents\n=\n[analyst], \ntasks\n=\n[analysis_task])\n\n\nresult \n=\n crew.kickoff()\n\n\n\n\nprint\n(result)\n\n\n\n\n\nError Handling\n\n\nThe reasoning process is designed to be robust, with error handling built in. If an error occurs during reasoning, the agent will proceed with executing the task without the reasoning plan. This ensures that tasks can still be executed even if the reasoning process fails.\n\n\nHere’s how to handle potential errors in your code:",
327 "metadata": {
328 "source": "https://docs.crewai.com/en/concepts/reasoning",
329 "title": "Reasoning - CrewAI"
330 }
331 },
332 {
333 "id": "03552db5-3f05-4f3a-a925-724d44ee9ed2",
334 "source": "https://docs.crewai.com/en/concepts/reasoning",
335 "title": "Reasoning - CrewAI",
336 "chunk_index": 5,
337 "text": "Here’s how to handle potential errors in your code:\n\n\nCopy\nAsk AI\nfrom\n crewai \nimport\n Agent, Task\n\n\nimport\n logging\n\n\n\n\n# Set up logging to capture any reasoning errors\n\n\nlogging.basicConfig(\nlevel\n=\nlogging.\nINFO\n)\n\n\n\n\n# Create an agent with reasoning enabled\n\n\nagent \n=\n Agent(\n\n\n role\n=\n\"Data Analyst\"\n,\n\n\n goal\n=\n\"Analyze data and provide insights\"\n,\n\n\n reasoning\n=\nTrue\n,\n\n\n max_reasoning_attempts\n=\n3\n\n\n)\n\n\n\n\n# Create a task\n\n\ntask \n=\n Task(\n\n\n description\n=\n\"Analyze the provided sales data and identify key trends.\"\n,\n\n\n expected_output\n=\n\"A report highlighting the top 3 sales trends.\"\n,\n\n\n agent\n=\nagent\n\n\n)\n\n\n\n\n# Execute the task\n\n\n# If an error occurs during reasoning, it will be logged and execution will continue\n\n\nresult \n=\n agent.execute_task(task)\n\n\n\n\n\nExample Reasoning Output\n\n\nHere’s an example of what a reasoning plan might look like for a data analysis task:\n\n\nCopy\nAsk AI\nTask: Analyze the provided sales data and identify key trends.",
338 "metadata": {
339 "source": "https://docs.crewai.com/en/concepts/reasoning",
340 "title": "Reasoning - CrewAI"
341 }
342 },
343 {
344 "id": "86c376b2-c82f-43d9-acd7-032163d0933b",
345 "source": "https://docs.crewai.com/en/concepts/reasoning",
346 "title": "Reasoning - CrewAI",
347 "chunk_index": 6,
348 "text": "\nExample Reasoning Output\n\n\nHere’s an example of what a reasoning plan might look like for a data analysis task:\n\n\nCopy\nAsk AI\nTask: Analyze the provided sales data and identify key trends.\n\n\n\n\nReasoning Plan:\n\n\nI'll analyze the sales data to identify the top 3 trends.\n\n\n\n\n1. Understanding of the task:\n\n\n I need to analyze sales data to identify key trends that would be valuable for business decision-making.\n\n\n\n\n2. Key steps I'll take:\n\n\n - First, I'll examine the data structure to understand what fields are available\n\n\n - Then I'll perform exploratory data analysis to identify patterns\n\n\n - Next, I'll analyze sales by time periods to identify temporal trends\n\n\n - I'll also analyze sales by product categories and customer segments\n\n\n - Finally, I'll identify the top 3 most significant trends\n\n\n\n\n3. Approach to challenges:\n\n\n - If the data has missing values, I'll decide whether to fill or filter them",
349 "metadata": {
350 "source": "https://docs.crewai.com/en/concepts/reasoning",
351 "title": "Reasoning - CrewAI"
352 }
353 },
354 {
355 "id": "11292abd-7566-4063-a23d-f80c7f017bf6",
356 "source": "https://docs.crewai.com/en/concepts/reasoning",
357 "title": "Reasoning - CrewAI",
358 "chunk_index": 7,
359 "text": "- Finally, I'll identify the top 3 most significant trends\n\n\n\n\n3. Approach to challenges:\n\n\n - If the data has missing values, I'll decide whether to fill or filter them\n\n\n - If the data has outliers, I'll investigate whether they're valid data points or errors\n\n\n - If trends aren't immediately obvious, I'll apply statistical methods to uncover patterns\n\n\n\n\n4. Use of available tools:\n\n\n - I'll use data analysis tools to explore and visualize the data\n\n\n - I'll use statistical tools to identify significant patterns\n\n\n - I'll use knowledge retrieval to access relevant information about sales analysis\n\n\n\n\n5. Expected outcome:\n\n\n A concise report highlighting the top 3 sales trends with supporting evidence from the data.\n\n\n\n\nREADY: I am ready to execute the task.",
360 "metadata": {
361 "source": "https://docs.crewai.com/en/concepts/reasoning",
362 "title": "Reasoning - CrewAI"
363 }
364 },
365 {
366 "id": "3b3ebb6b-eabf-4d79-8858-e514f36d3740",
367 "source": "https://docs.crewai.com/en/concepts/reasoning",
368 "title": "Reasoning - CrewAI",
369 "chunk_index": 8,
370 "text": "5. Expected outcome:\n\n\n A concise report highlighting the top 3 sales trends with supporting evidence from the data.\n\n\n\n\nREADY: I am ready to execute the task.\n\n\n\n\nThis reasoning plan helps the agent organize its approach to the task, consider potential challenges, and ensure it delivers the expected output.\nAssistant\nResponses are generated using AI and may contain mistakes.\nWas this page helpful?\nYes\nNo\nMemory\nPlanning\nwebsite\nx\ngithub\nlinkedin\nyoutube\nreddit\nPowered by Mintlify\nOn this page\nOverview\nUsage\nHow It Works\nConfiguration Options\nExample\nError Handling\nExample Reasoning Output\nCore Concepts\nReasoning\nCopy page\nLearn how to enable and use agent reasoning to improve task execution.\n\nOverview\n\n\nAgent reasoning is a feature that allows agents to reflect on a task and create a plan before execution. This helps agents approach tasks more methodically and ensures they’re ready to perform the assigned work.\n\n\n\nUsage",
371 "metadata": {
372 "source": "https://docs.crewai.com/en/concepts/reasoning",
373 "title": "Reasoning - CrewAI"
374 }
375 },
376 {
377 "id": "8353f782-9e8c-4d08-8a2d-a16cf80de9d0",
378 "source": "https://docs.crewai.com/en/concepts/reasoning",
379 "title": "Reasoning - CrewAI",
380 "chunk_index": 9,
381 "text": "\nUsage\n\n\nTo enable reasoning for an agent, simply set \nreasoning=True\n when creating the agent:\n\n\nCopy\nAsk AI\nfrom\n crewai \nimport\n Agent\n\n\n\n\nagent \n=\n Agent(\n\n\n role\n=\n\"Data Analyst\"\n,\n\n\n goal\n=\n\"Analyze complex datasets and provide insights\"\n,\n\n\n backstory\n=\n\"You are an experienced data analyst with expertise in finding patterns in complex data.\"\n,\n\n\n reasoning\n=\nTrue\n, \n# Enable reasoning\n\n\n max_reasoning_attempts\n=\n3\n # Optional: Set a maximum number of reasoning attempts\n\n\n)\n\n\n\n\n\nHow It Works\n\n\nWhen reasoning is enabled, before executing a task, the agent will:\n\n\n\n\nReflect on the task and create a detailed plan\n\n\nEvaluate whether it’s ready to execute the task\n\n\nRefine the plan as necessary until it’s ready or max_reasoning_attempts is reached\n\n\nInject the reasoning plan into the task description before execution\n\n\n\n\nThis process helps the agent break down complex tasks into manageable steps and identify potential challenges before starting.",
382 "metadata": {
383 "source": "https://docs.crewai.com/en/concepts/reasoning",
384 "title": "Reasoning - CrewAI"
385 }
386 },
387 {
388 "id": "a96a91c9-a96d-4ca7-8d89-d85b7afd0b7b",
389 "source": "https://docs.crewai.com/en/concepts/reasoning",
390 "title": "Reasoning - CrewAI",
391 "chunk_index": 10,
392 "text": "This process helps the agent break down complex tasks into manageable steps and identify potential challenges before starting.\n\n\n\nConfiguration Options\n\n\n\nreasoning\nbool\ndefault:\n\"False\"\nEnable or disable reasoning\n\n\n\nmax_reasoning_attempts\nint\ndefault:\n\"None\"\nMaximum number of attempts to refine the plan before proceeding with execution. If None (default), the agent will continue refining until it’s ready.\n\n\n\nExample\n\n\nHere’s a complete example:\n\n\nCopy\nAsk AI\nfrom\n crewai \nimport\n Agent, Task, Crew\n\n\n\n\n# Create an agent with reasoning enabled\n\n\nanalyst \n=\n Agent(\n\n\n role\n=\n\"Data Analyst\"\n,\n\n\n goal\n=\n\"Analyze data and provide insights\"\n,\n\n\n backstory\n=\n\"You are an expert data analyst.\"\n,\n\n\n reasoning\n=\nTrue\n,\n\n\n max_reasoning_attempts\n=\n3\n # Optional: Set a limit on reasoning attempts\n\n\n)\n\n\n\n\n# Create a task\n\n\nanalysis_task \n=\n Task(\n\n\n description\n=\n\"Analyze the provided sales data and identify key trends.\"\n,",
393 "metadata": {
394 "source": "https://docs.crewai.com/en/concepts/reasoning",
395 "title": "Reasoning - CrewAI"
396 }
397 },
398 {
399 "id": "3ba42849-c1be-461d-a041-b5bea89c1b88",
400 "source": "https://docs.crewai.com/en/concepts/reasoning",
401 "title": "Reasoning - CrewAI",
402 "chunk_index": 11,
403 "text": ")\n\n\n\n\n# Create a task\n\n\nanalysis_task \n=\n Task(\n\n\n description\n=\n\"Analyze the provided sales data and identify key trends.\"\n,\n\n\n expected_output\n=\n\"A report highlighting the top 3 sales trends.\"\n,\n\n\n agent\n=\nanalyst\n\n\n)\n\n\n\n\n# Create a crew and run the task\n\n\ncrew \n=\n Crew(\nagents\n=\n[analyst], \ntasks\n=\n[analysis_task])\n\n\nresult \n=\n crew.kickoff()\n\n\n\n\nprint\n(result)\n\n\n\n\n\nError Handling\n\n\nThe reasoning process is designed to be robust, with error handling built in. If an error occurs during reasoning, the agent will proceed with executing the task without the reasoning plan. This ensures that tasks can still be executed even if the reasoning process fails.\n\n\nHere’s how to handle potential errors in your code:\n\n\nCopy\nAsk AI\nfrom\n crewai \nimport\n Agent, Task\n\n\nimport\n logging\n\n\n\n\n# Set up logging to capture any reasoning errors\n\n\nlogging.basicConfig(\nlevel\n=\nlogging.\nINFO\n)\n\n\n\n\n# Create an agent with reasoning enabled\n\n\nagent \n=\n Agent(\n\n\n role\n=\n\"Data Analyst\"\n,",
404 "metadata": {
405 "source": "https://docs.crewai.com/en/concepts/reasoning",
406 "title": "Reasoning - CrewAI"
407 }
408 },
409 {
410 "id": "0116ac8c-224c-4505-80ee-01d8cdee8df7",
411 "source": "https://docs.crewai.com/en/concepts/reasoning",
412 "title": "Reasoning - CrewAI",
413 "chunk_index": 12,
414 "text": "# Set up logging to capture any reasoning errors\n\n\nlogging.basicConfig(\nlevel\n=\nlogging.\nINFO\n)\n\n\n\n\n# Create an agent with reasoning enabled\n\n\nagent \n=\n Agent(\n\n\n role\n=\n\"Data Analyst\"\n,\n\n\n goal\n=\n\"Analyze data and provide insights\"\n,\n\n\n reasoning\n=\nTrue\n,\n\n\n max_reasoning_attempts\n=\n3\n\n\n)\n\n\n\n\n# Create a task\n\n\ntask \n=\n Task(\n\n\n description\n=\n\"Analyze the provided sales data and identify key trends.\"\n,\n\n\n expected_output\n=\n\"A report highlighting the top 3 sales trends.\"\n,\n\n\n agent\n=\nagent\n\n\n)\n\n\n\n\n# Execute the task\n\n\n# If an error occurs during reasoning, it will be logged and execution will continue\n\n\nresult \n=\n agent.execute_task(task)\n\n\n\n\n\nExample Reasoning Output\n\n\nHere’s an example of what a reasoning plan might look like for a data analysis task:\n\n\nCopy\nAsk AI\nTask: Analyze the provided sales data and identify key trends.\n\n\n\n\nReasoning Plan:\n\n\nI'll analyze the sales data to identify the top 3 trends.\n\n\n\n\n1. Understanding of the task:",
415 "metadata": {
416 "source": "https://docs.crewai.com/en/concepts/reasoning",
417 "title": "Reasoning - CrewAI"
418 }
419 },
420 {
421 "id": "7d6129af-b188-4e9b-ac74-edc6160d4159",
422 "source": "https://docs.crewai.com/en/concepts/reasoning",
423 "title": "Reasoning - CrewAI",
424 "chunk_index": 13,
425 "text": "Copy\nAsk AI\nTask: Analyze the provided sales data and identify key trends.\n\n\n\n\nReasoning Plan:\n\n\nI'll analyze the sales data to identify the top 3 trends.\n\n\n\n\n1. Understanding of the task:\n\n\n I need to analyze sales data to identify key trends that would be valuable for business decision-making.\n\n\n\n\n2. Key steps I'll take:\n\n\n - First, I'll examine the data structure to understand what fields are available\n\n\n - Then I'll perform exploratory data analysis to identify patterns\n\n\n - Next, I'll analyze sales by time periods to identify temporal trends\n\n\n - I'll also analyze sales by product categories and customer segments\n\n\n - Finally, I'll identify the top 3 most significant trends\n\n\n\n\n3. Approach to challenges:\n\n\n - If the data has missing values, I'll decide whether to fill or filter them\n\n\n - If the data has outliers, I'll investigate whether they're valid data points or errors",
426 "metadata": {
427 "source": "https://docs.crewai.com/en/concepts/reasoning",
428 "title": "Reasoning - CrewAI"
429 }
430 },
431 {
432 "id": "7515850d-fd1f-4754-9853-2e20923aaba8",
433 "source": "https://docs.crewai.com/en/concepts/reasoning",
434 "title": "Reasoning - CrewAI",
435 "chunk_index": 14,
436 "text": "- If the data has missing values, I'll decide whether to fill or filter them\n\n\n - If the data has outliers, I'll investigate whether they're valid data points or errors\n\n\n - If trends aren't immediately obvious, I'll apply statistical methods to uncover patterns\n\n\n\n\n4. Use of available tools:\n\n\n - I'll use data analysis tools to explore and visualize the data\n\n\n - I'll use statistical tools to identify significant patterns\n\n\n - I'll use knowledge retrieval to access relevant information about sales analysis\n\n\n\n\n5. Expected outcome:\n\n\n A concise report highlighting the top 3 sales trends with supporting evidence from the data.\n\n\n\n\nREADY: I am ready to execute the task.",
437 "metadata": {
438 "source": "https://docs.crewai.com/en/concepts/reasoning",
439 "title": "Reasoning - CrewAI"
440 }
441 },
442 {
443 "id": "0a0c6e63-8b04-42c1-8ded-60adf25086e5",
444 "source": "https://docs.crewai.com/en/concepts/reasoning",
445 "title": "Reasoning - CrewAI",
446 "chunk_index": 15,
447 "text": "5. Expected outcome:\n\n\n A concise report highlighting the top 3 sales trends with supporting evidence from the data.\n\n\n\n\nREADY: I am ready to execute the task.\n\n\n\n\nThis reasoning plan helps the agent organize its approach to the task, consider potential challenges, and ensure it delivers the expected output.\nAssistant\nResponses are generated using AI and may contain mistakes.\nWas this page helpful?\nYes\nNo\nMemory\nPlanning\nwebsite\nx\ngithub\nlinkedin\nyoutube\nreddit\nPowered by Mintlify\nOn this page\nOverview\nUsage\nHow It Works\nConfiguration Options\nExample\nError Handling\nExample Reasoning Output",
448 "metadata": {
449 "source": "https://docs.crewai.com/en/concepts/reasoning",
450 "title": "Reasoning - CrewAI"
451 }
452 },
453 {
454 "id": "1e78eb85-4571-4287-9eeb-13a468f56f8d",
455 "source": "https://docs.crewai.com/en/observability/portkey",
456 "title": "Portkey Integration - CrewAI",
457 "chunk_index": 0,
458 "text": "CrewAI\n home page\nEnglish\nSearch...\n⌘\nK\nAsk AI\nStart Cloud Trial\ncrewAIInc\n/\ncrewAI\ncrewAIInc\n/\ncrewAI\nSearch...\nNavigation\nObservability\nPortkey Integration\nDocumentation\nEnterprise\nAPI Reference\nExamples\nWebsite\nForum\nCrew GPT\nGet Help\nReleases\nGet Started\nIntroduction\nInstallation\nQuickstart\nGuides\nStrategy\nAgents\nCrews\nFlows\nAdvanced\nCore Concepts\nAgents\nTasks\nCrews\nFlows\nKnowledge\nLLMs\nProcesses\nCollaboration\nTraining\nMemory\nReasoning\nPlanning\nTesting\nCLI\nTools\nEvent Listeners\nMCP Integration\nMCP Servers as Tools in CrewAI\nStdio Transport\nSSE Transport\nStreamable HTTP Transport\nConnecting to Multiple MCP Servers\nMCP Security Considerations\nTools\nTools Overview\nFile & Document\nWeb Scraping & Browsing\nSearch & Research\nDatabase & Data\nAI & Machine Learning\nCloud & Storage\nAutomation & Integration\nObservability\nOverview\nAgentOps Integration\nArize Phoenix\nLangfuse Integration\nLangtrace Integration\nMaxim Integration\nMLflow Integration\nOpenLIT Integration\nOpik Integration",
459 "metadata": {
460 "source": "https://docs.crewai.com/en/observability/portkey",
461 "title": "Portkey Integration - CrewAI"
462 }
463 },
464 {
465 "id": "7b884544-3c59-4e35-8d70-cfe6bfa0c785",
466 "source": "https://docs.crewai.com/en/observability/portkey",
467 "title": "Portkey Integration - CrewAI",
468 "chunk_index": 1,
469 "text": "Automation & Integration\nObservability\nOverview\nAgentOps Integration\nArize Phoenix\nLangfuse Integration\nLangtrace Integration\nMaxim Integration\nMLflow Integration\nOpenLIT Integration\nOpik Integration\nPatronus AI Evaluation\nPortkey Integration\nWeave Integration\nLearn\nOverview\nStrategic LLM Selection Guide\nConditional Tasks\nCoding Agents\nCreate Custom Tools\nCustom LLM Implementation\nCustom Manager Agent\nCustomize Agents\nImage Generation with DALL-E\nForce Tool Output as Result\nHierarchical Process\nHuman Input on Execution\nKickoff Crew Asynchronously\nKickoff Crew for Each\nConnect to any LLM\nUsing Multimodal Agents\nReplay Tasks from Latest Crew Kickoff\nSequential Processes\nUsing Annotations in crew.py\nTelemetry\nTelemetry\nObservability\nPortkey Integration\nCopy page\nHow to use Portkey with CrewAI",
470 "metadata": {
471 "source": "https://docs.crewai.com/en/observability/portkey",
472 "title": "Portkey Integration - CrewAI"
473 }
474 },
475 {
476 "id": "fc0c84e6-34cb-4ecc-aa4d-1c743c577af4",
477 "source": "https://docs.crewai.com/en/observability/portkey",
478 "title": "Portkey Integration - CrewAI",
479 "chunk_index": 2,
480 "text": "\nIntroduction\n\n\nPortkey enhances CrewAI with production-readiness features, turning your experimental agent crews into robust systems by providing:\n\n\n\n\nComplete observability\n of every agent step, tool use, and interaction\n\n\nBuilt-in reliability\n with fallbacks, retries, and load balancing\n\n\nCost tracking and optimization\n to manage your AI spend\n\n\nAccess to 200+ LLMs\n through a single integration\n\n\nGuardrails\n to keep agent behavior safe and compliant\n\n\nVersion-controlled prompts\n for consistent agent performance\n\n\n\n\n\nInstallation & Setup\n\n\n1\nInstall the required packages\nCopy\nAsk AI\npip\n install\n -U\n crewai\n portkey-ai",
481 "metadata": {
482 "source": "https://docs.crewai.com/en/observability/portkey",
483 "title": "Portkey Integration - CrewAI"
484 }
485 },
486 {
487 "id": "dca237af-9937-4b01-b82c-ae4dc989b25c",
488 "source": "https://docs.crewai.com/en/observability/portkey",
489 "title": "Portkey Integration - CrewAI",
490 "chunk_index": 3,
491 "text": "Version-controlled prompts\n for consistent agent performance\n\n\n\n\n\nInstallation & Setup\n\n\n1\nInstall the required packages\nCopy\nAsk AI\npip\n install\n -U\n crewai\n portkey-ai\n\n\nGenerate API Key\nCreate a Portkey API key with optional budget/rate limits from the \nPortkey dashboard\n. You can also attach configurations for reliability, caching, and more to this key. More on this later.\n3\nConfigure CrewAI with Portkey\nThe integration is simple - you just need to update the LLM configuration in your CrewAI setup:\nCopy\nAsk AI\nfrom\n crewai \nimport\n LLM\n\n\nfrom\n portkey_ai \nimport\n createHeaders, \nPORTKEY_GATEWAY_URL\n\n\n\n\n# Create an LLM instance with Portkey integration\n\n\ngpt_llm \n=\n LLM(\n\n\n model\n=\n\"gpt-4o\"\n,\n\n\n base_url\n=\nPORTKEY_GATEWAY_URL\n,\n\n\n api_key\n=\n\"dummy\"\n, \n# We are using a Virtual key, so this is a placeholder\n\n\n extra_headers\n=\ncreateHeaders(\n\n\n api_key\n=\n\"YOUR_PORTKEY_API_KEY\"\n,\n\n\n virtual_key\n=\n\"YOUR_LLM_VIRTUAL_KEY\"\n,",
492 "metadata": {
493 "source": "https://docs.crewai.com/en/observability/portkey",
494 "title": "Portkey Integration - CrewAI"
495 }
496 },
497 {
498 "id": "73c59de8-2f08-49eb-8d1b-07c463b6833d",
499 "source": "https://docs.crewai.com/en/observability/portkey",
500 "title": "Portkey Integration - CrewAI",
501 "chunk_index": 4,
502 "text": "extra_headers\n=\ncreateHeaders(\n\n\n api_key\n=\n\"YOUR_PORTKEY_API_KEY\"\n,\n\n\n virtual_key\n=\n\"YOUR_LLM_VIRTUAL_KEY\"\n,\n\n\n trace_id\n=\n\"unique-trace-id\"\n, \n# Optional, for request tracing\n\n\n )\n\n\n)\n\n\n\n\n#Use them in your Crew Agents like this:\n\n\n\n\n\t@agent\n\n\n\tdef\n lead_market_analyst\n(\nself\n) -> Agent:\n\n\n\t\treturn\n Agent(\n\n\n\t\t\tconfig\n=\nself\n.agents_config[\n'lead_market_analyst'\n],\n\n\n\t\t\tverbose\n=\nTrue\n,\n\n\n\t\t\tmemory\n=\nFalse\n,\n\n\n\t\t\tllm\n=\ngpt_llm\n\n\n\t\t)\n\n\n\n\nWhat are Virtual Keys?\n Virtual keys in Portkey securely store your LLM provider API keys (OpenAI, Anthropic, etc.) in an encrypted vault. They allow for easier key rotation and budget management. \nLearn more about virtual keys here\n.\n\n\n\nProduction Features\n\n\n\n1. Enhanced Observability\n\n\nPortkey provides comprehensive observability for your CrewAI agents, helping you understand exactly what’s happening during each execution.",
503 "metadata": {
504 "source": "https://docs.crewai.com/en/observability/portkey",
505 "title": "Portkey Integration - CrewAI"
506 }
507 },
508 {
509 "id": "d1a2bd8a-4f56-4f5e-a3b7-5f5356fb14f0",
510 "source": "https://docs.crewai.com/en/observability/portkey",
511 "title": "Portkey Integration - CrewAI",
512 "chunk_index": 5,
513 "text": "\nProduction Features\n\n\n\n1. Enhanced Observability\n\n\nPortkey provides comprehensive observability for your CrewAI agents, helping you understand exactly what’s happening during each execution.\n\n\nTraces\nLogs\nMetrics & Dashboards\nMetadata Filtering\nTraces provide a hierarchical view of your crew’s execution, showing the sequence of LLM calls, tool invocations, and state transitions.\nCopy\nAsk AI\n# Add trace_id to enable hierarchical tracing in Portkey\n\n\nportkey_llm \n=\n LLM(\n\n\n model\n=\n\"gpt-4o\"\n,\n\n\n base_url\n=\nPORTKEY_GATEWAY_URL\n,\n\n\n api_key\n=\n\"dummy\"\n,\n\n\n extra_headers\n=\ncreateHeaders(\n\n\n api_key\n=\n\"YOUR_PORTKEY_API_KEY\"\n,\n\n\n virtual_key\n=\n\"YOUR_OPENAI_VIRTUAL_KEY\"\n,\n\n\n trace_id\n=\n\"unique-session-id\"\n # Add unique trace ID\n\n\n )\n\n\n)\n\n\nTraces provide a hierarchical view of your crew’s execution, showing the sequence of LLM calls, tool invocations, and state transitions.\nCopy\nAsk AI\n# Add trace_id to enable hierarchical tracing in Portkey",
514 "metadata": {
515 "source": "https://docs.crewai.com/en/observability/portkey",
516 "title": "Portkey Integration - CrewAI"
517 }
518 },
519 {
520 "id": "b9a9e74b-4273-48b1-9b1b-483cd49e5236",
521 "source": "https://docs.crewai.com/en/observability/portkey",
522 "title": "Portkey Integration - CrewAI",
523 "chunk_index": 6,
524 "text": "portkey_llm \n=\n LLM(\n\n\n model\n=\n\"gpt-4o\"\n,\n\n\n base_url\n=\nPORTKEY_GATEWAY_URL\n,\n\n\n api_key\n=\n\"dummy\"\n,\n\n\n extra_headers\n=\ncreateHeaders(\n\n\n api_key\n=\n\"YOUR_PORTKEY_API_KEY\"\n,\n\n\n virtual_key\n=\n\"YOUR_OPENAI_VIRTUAL_KEY\"\n,\n\n\n trace_id\n=\n\"unique-session-id\"\n # Add unique trace ID\n\n\n )\n\n\n)\n\n\nPortkey logs every interaction with LLMs, including:\n\n\nComplete request and response payloads\n\n\nLatency and token usage metrics\n\n\nCost calculations\n\n\nTool calls and function executions\n\n\nAll logs can be filtered by metadata, trace IDs, models, and more, making it easy to debug specific crew runs.\nPortkey provides built-in dashboards that help you:\n\n\nTrack cost and token usage across all crew runs\n\n\nAnalyze performance metrics like latency and success rates\n\n\nIdentify bottlenecks in your agent workflows\n\n\nCompare different crew configurations and LLMs",
525 "metadata": {
526 "source": "https://docs.crewai.com/en/observability/portkey",
527 "title": "Portkey Integration - CrewAI"
528 }
529 },
530 {
531 "id": "3d0eb246-23dc-4355-ae99-0f8610217b3b",
532 "source": "https://docs.crewai.com/en/observability/portkey",
533 "title": "Portkey Integration - CrewAI",
534 "chunk_index": 7,
535 "text": "Analyze performance metrics like latency and success rates\n\n\nIdentify bottlenecks in your agent workflows\n\n\nCompare different crew configurations and LLMs\n\n\nYou can filter and segment all metrics by custom metadata to analyze specific crew types, user groups, or use cases.\nAdd custom metadata to your CrewAI LLM configuration to enable powerful filtering and segmentation:\nCopy\nAsk AI\nportkey_llm \n=\n LLM(\n\n\n model\n=\n\"gpt-4o\"\n,\n\n\n base_url\n=\nPORTKEY_GATEWAY_URL\n,\n\n\n api_key\n=\n\"dummy\"\n,\n\n\n extra_headers\n=\ncreateHeaders(\n\n\n api_key\n=\n\"YOUR_PORTKEY_API_KEY\"\n,\n\n\n virtual_key\n=\n\"YOUR_OPENAI_VIRTUAL_KEY\"\n,\n\n\n metadata\n=\n{\n\n\n \"crew_type\"\n: \n\"research_crew\"\n,\n\n\n \"environment\"\n: \n\"production\"\n,\n\n\n \"_user\"\n: \n\"user_123\"\n, \n# Special _user field for user analytics\n\n\n \"request_source\"\n: \n\"mobile_app\"\n\n\n }\n\n\n )\n\n\n)",
536 "metadata": {
537 "source": "https://docs.crewai.com/en/observability/portkey",
538 "title": "Portkey Integration - CrewAI"
539 }
540 },
541 {
542 "id": "ec8dbabd-5b5f-4895-a606-4aa6034b9d60",
543 "source": "https://docs.crewai.com/en/observability/portkey",
544 "title": "Portkey Integration - CrewAI",
545 "chunk_index": 8,
546 "text": "\"environment\"\n: \n\"production\"\n,\n\n\n \"_user\"\n: \n\"user_123\"\n, \n# Special _user field for user analytics\n\n\n \"request_source\"\n: \n\"mobile_app\"\n\n\n }\n\n\n )\n\n\n)\n\n\nThis metadata can be used to filter logs, traces, and metrics on the Portkey dashboard, allowing you to analyze specific crew runs, users, or environments.\n\n\n\n2. Reliability - Keep Your Crews Running Smoothly\n\n\nWhen running crews in production, things can go wrong - API rate limits, network issues, or provider outages. Portkey’s reliability features ensure your agents keep running smoothly even when problems occur.\n\n\nIt’s simple to enable fallback in your CrewAI setup by using a Portkey Config:\n\n\nCopy\nAsk AI\nfrom\n crewai \nimport\n LLM\n\n\nfrom\n portkey_ai \nimport\n createHeaders, \nPORTKEY_GATEWAY_URL\n\n\n\n\n# Create LLM with fallback configuration\n\n\nportkey_llm \n=\n LLM(\n\n\n model\n=\n\"gpt-4o\"\n,\n\n\n max_tokens\n=\n1000\n,\n\n\n base_url\n=\nPORTKEY_GATEWAY_URL\n,\n\n\n api_key\n=\n\"dummy\"\n,",
547 "metadata": {
548 "source": "https://docs.crewai.com/en/observability/portkey",
549 "title": "Portkey Integration - CrewAI"
550 }
551 },
552 {
553 "id": "90513a18-9659-463f-b382-b24ab31b2177",
554 "source": "https://docs.crewai.com/en/observability/portkey",
555 "title": "Portkey Integration - CrewAI",
556 "chunk_index": 9,
557 "text": "# Create LLM with fallback configuration\n\n\nportkey_llm \n=\n LLM(\n\n\n model\n=\n\"gpt-4o\"\n,\n\n\n max_tokens\n=\n1000\n,\n\n\n base_url\n=\nPORTKEY_GATEWAY_URL\n,\n\n\n api_key\n=\n\"dummy\"\n,\n\n\n extra_headers\n=\ncreateHeaders(\n\n\n api_key\n=\n\"YOUR_PORTKEY_API_KEY\"\n,\n\n\n config\n=\n{\n\n\n \"strategy\"\n: {\n\n\n \"mode\"\n: \n\"fallback\"\n\n\n },\n\n\n \"targets\"\n: [\n\n\n {\n\n\n \"provider\"\n: \n\"openai\"\n,\n\n\n \"api_key\"\n: \n\"YOUR_OPENAI_API_KEY\"\n,\n\n\n \"override_params\"\n: {\n\"model\"\n: \n\"gpt-4o\"\n}\n\n\n },\n\n\n {\n\n\n \"provider\"\n: \n\"anthropic\"\n,\n\n\n \"api_key\"\n: \n\"YOUR_ANTHROPIC_API_KEY\"\n,\n\n\n \"override_params\"\n: {\n\"model\"\n: \n\"claude-3-opus-20240229\"\n}\n\n\n }\n\n\n ]\n\n\n }\n\n\n )\n\n\n)\n\n\n\n\n# Use this LLM configuration with your agents",
558 "metadata": {
559 "source": "https://docs.crewai.com/en/observability/portkey",
560 "title": "Portkey Integration - CrewAI"
561 }
562 },
563 {
564 "id": "88197d87-051c-4458-8da1-d079143aa232",
565 "source": "https://docs.crewai.com/en/observability/portkey",
566 "title": "Portkey Integration - CrewAI",
567 "chunk_index": 10,
568 "text": "\"override_params\"\n: {\n\"model\"\n: \n\"claude-3-opus-20240229\"\n}\n\n\n }\n\n\n ]\n\n\n }\n\n\n )\n\n\n)\n\n\n\n\n# Use this LLM configuration with your agents\n\n\n\n\nThis configuration will automatically try Claude if the GPT-4o request fails, ensuring your crew can continue operating.",
569 "metadata": {
570 "source": "https://docs.crewai.com/en/observability/portkey",
571 "title": "Portkey Integration - CrewAI"
572 }
573 },
574 {
575 "id": "eeaa78c1-5f9a-43b3-af53-1a4f5d6db956",
576 "source": "https://docs.crewai.com/en/observability/portkey",
577 "title": "Portkey Integration - CrewAI",
578 "chunk_index": 11,
579 "text": "}\n\n\n )\n\n\n)\n\n\n\n\n# Use this LLM configuration with your agents\n\n\n\n\nThis configuration will automatically try Claude if the GPT-4o request fails, ensuring your crew can continue operating.\n\n\nAutomatic Retries\nHandles temporary failures automatically. If an LLM call fails, Portkey will retry the same request for the specified number of times - perfect for rate limits or network blips.\nRequest Timeouts\nPrevent your agents from hanging. Set timeouts to ensure you get responses (or can fail gracefully) within your required timeframes.\nConditional Routing\nSend different requests to different providers. Route complex reasoning to GPT-4, creative tasks to Claude, and quick responses to Gemini based on your needs.\nFallbacks\nKeep running even if your primary provider fails. Automatically switch to backup providers to maintain availability.\nLoad Balancing\nSpread requests across multiple API keys or providers. Great for high-volume crew operations and staying within rate limits.",
580 "metadata": {
581 "source": "https://docs.crewai.com/en/observability/portkey",
582 "title": "Portkey Integration - CrewAI"
583 }
584 },
585 {
586 "id": "b4a06534-031a-4d8d-9412-c48aefb455f6",
587 "source": "https://docs.crewai.com/en/observability/portkey",
588 "title": "Portkey Integration - CrewAI",
589 "chunk_index": 12,
590 "text": "\n3. Prompting in CrewAI\n\n\nPortkey’s Prompt Engineering Studio helps you create, manage, and optimize the prompts used in your CrewAI agents. Instead of hardcoding prompts or instructions, use Portkey’s prompt rendering API to dynamically fetch and apply your versioned prompts.\n\n\nManage prompts in Portkey's Prompt Library\n\n\nPrompt Playground\nUsing Prompt Templates\nPrompt Versioning\nMustache Templating for variables\nPrompt Playground is a place to compare, test and deploy perfect prompts for your AI application. It’s where you experiment with different models, test variables, compare outputs, and refine your prompt engineering strategy before deploying to production. It allows you to:\n\n\nIteratively develop prompts before using them in your agents\n\n\nTest prompts with different variables and models\n\n\nCompare outputs between different prompt versions\n\n\nCollaborate with team members on prompt development",
591 "metadata": {
592 "source": "https://docs.crewai.com/en/observability/portkey",
593 "title": "Portkey Integration - CrewAI"
594 }
595 },
596 {
597 "id": "446f0865-7a26-4707-a7d8-fa1c4713f8fe",
598 "source": "https://docs.crewai.com/en/observability/portkey",
599 "title": "Portkey Integration - CrewAI",
600 "chunk_index": 13,
601 "text": "Test prompts with different variables and models\n\n\nCompare outputs between different prompt versions\n\n\nCollaborate with team members on prompt development\n\n\nThis visual environment makes it easier to craft effective prompts for each step in your CrewAI agents’ workflow.\nPrompt Playground is a place to compare, test and deploy perfect prompts for your AI application. It’s where you experiment with different models, test variables, compare outputs, and refine your prompt engineering strategy before deploying to production. It allows you to:\n\n\nIteratively develop prompts before using them in your agents\n\n\nTest prompts with different variables and models\n\n\nCompare outputs between different prompt versions\n\n\nCollaborate with team members on prompt development",
602 "metadata": {
603 "source": "https://docs.crewai.com/en/observability/portkey",
604 "title": "Portkey Integration - CrewAI"
605 }
606 },
607 {
608 "id": "c099e789-0061-45d9-9eed-6bebc93cfd07",
609 "source": "https://docs.crewai.com/en/observability/portkey",
610 "title": "Portkey Integration - CrewAI",
611 "chunk_index": 14,
612 "text": "Test prompts with different variables and models\n\n\nCompare outputs between different prompt versions\n\n\nCollaborate with team members on prompt development\n\n\nThis visual environment makes it easier to craft effective prompts for each step in your CrewAI agents’ workflow.\nThe Prompt Render API retrieves your prompt templates with all parameters configured:\nCopy\nAsk AI\nfrom\n crewai \nimport\n Agent, \nLLM\n\n\nfrom\n portkey_ai \nimport\n createHeaders, \nPORTKEY_GATEWAY_URL\n, Portkey\n\n\n\n\n# Initialize Portkey admin client\n\n\nportkey_admin \n=\n Portkey(\napi_key\n=\n\"YOUR_PORTKEY_API_KEY\"\n)\n\n\n\n\n# Retrieve prompt using the render API\n\n\nprompt_data \n=\n portkey_client.prompts.render(\n\n\n prompt_id\n=\n\"YOUR_PROMPT_ID\"\n,\n\n\n variables\n=\n{\n\n\n \"agent_role\"\n: \n\"Senior Research Scientist\"\n,\n\n\n }\n\n\n)\n\n\n\n\nbackstory_agent_prompt\n=\nprompt_data.data.messages[\n0\n][\n\"content\"\n]\n\n\n\n\n\n\n# Set up LLM with Portkey integration\n\n\nportkey_llm \n=\n LLM(\n\n\n model\n=\n\"gpt-4o\"\n,",
613 "metadata": {
614 "source": "https://docs.crewai.com/en/observability/portkey",
615 "title": "Portkey Integration - CrewAI"
616 }
617 },
618 {
619 "id": "89fa7056-e42a-4286-8b90-8084dd34ef3b",
620 "source": "https://docs.crewai.com/en/observability/portkey",
621 "title": "Portkey Integration - CrewAI",
622 "chunk_index": 15,
623 "text": "}\n\n\n)\n\n\n\n\nbackstory_agent_prompt\n=\nprompt_data.data.messages[\n0\n][\n\"content\"\n]\n\n\n\n\n\n\n# Set up LLM with Portkey integration\n\n\nportkey_llm \n=\n LLM(\n\n\n model\n=\n\"gpt-4o\"\n,\n\n\n base_url\n=\nPORTKEY_GATEWAY_URL\n,\n\n\n api_key\n=\n\"dummy\"\n,\n\n\n extra_headers\n=\ncreateHeaders(\n\n\n api_key\n=\n\"YOUR_PORTKEY_API_KEY\"\n,\n\n\n virtual_key\n=\n\"YOUR_OPENAI_VIRTUAL_KEY\"\n\n\n )\n\n\n)\n\n\n\n\n# Create agent using the rendered prompt\n\n\nresearcher \n=\n Agent(\n\n\n role\n=\n\"Senior Research Scientist\"\n,\n\n\n goal\n=\n\"Discover groundbreaking insights about the assigned topic\"\n,\n\n\n backstory\n=\nbackstory_agent, \n# Use the rendered prompt\n\n\n verbose\n=\nTrue\n,\n\n\n llm\n=\nportkey_llm\n\n\n)\n\n\nYou can:\n\n\nCreate multiple versions of the same prompt\n\n\nCompare performance between versions\n\n\nRoll back to previous versions if needed\n\n\nSpecify which version to use in your code:\n\n\nCopy\nAsk AI\n# Use a specific prompt version\n\n\nprompt_data \n=\n portkey_admin.prompts.render(",
624 "metadata": {
625 "source": "https://docs.crewai.com/en/observability/portkey",
626 "title": "Portkey Integration - CrewAI"
627 }
628 },
629 {
630 "id": "a4cf2636-997f-427e-8437-cffd5b6a1829",
631 "source": "https://docs.crewai.com/en/observability/portkey",
632 "title": "Portkey Integration - CrewAI",
633 "chunk_index": 16,
634 "text": "Roll back to previous versions if needed\n\n\nSpecify which version to use in your code:\n\n\nCopy\nAsk AI\n# Use a specific prompt version\n\n\nprompt_data \n=\n portkey_admin.prompts.render(\n\n\n prompt_id\n=\n\"YOUR_PROMPT_ID@version_number\"\n,\n\n\n variables\n=\n{\n\n\n \"agent_role\"\n: \n\"Senior Research Scientist\"\n,\n\n\n \"agent_goal\"\n: \n\"Discover groundbreaking insights\"\n\n\n }\n\n\n)\n\n\nPortkey prompts use Mustache-style templating for easy variable substitution:\nCopy\nAsk AI\nYou are a {{agent_role}} with expertise in {{domain}}.\n\n\n\n\nYour mission is to {{agent_goal}} by leveraging your knowledge\n\n\nand experience in the field.\n\n\n\n\nAlways maintain a {{tone}} tone and focus on providing {{focus_area}}.\n\n\nWhen rendering, simply pass the variables:\nCopy\nAsk AI\nprompt_data \n=\n portkey_admin.prompts.render(\n\n\n prompt_id\n=\n\"YOUR_PROMPT_ID\"\n,\n\n\n variables\n=\n{\n\n\n \"agent_role\"\n: \n\"Senior Research Scientist\"\n,\n\n\n \"domain\"\n: \n\"artificial intelligence\"\n,",
635 "metadata": {
636 "source": "https://docs.crewai.com/en/observability/portkey",
637 "title": "Portkey Integration - CrewAI"
638 }
639 },
640 {
641 "id": "595bb7f4-ad28-42b0-8fc3-4e94fa6d48e5",
642 "source": "https://docs.crewai.com/en/observability/portkey",
643 "title": "Portkey Integration - CrewAI",
644 "chunk_index": 17,
645 "text": "prompt_id\n=\n\"YOUR_PROMPT_ID\"\n,\n\n\n variables\n=\n{\n\n\n \"agent_role\"\n: \n\"Senior Research Scientist\"\n,\n\n\n \"domain\"\n: \n\"artificial intelligence\"\n,\n\n\n \"agent_goal\"\n: \n\"discover groundbreaking insights\"\n,\n\n\n \"tone\"\n: \n\"professional\"\n,\n\n\n \"focus_area\"\n: \n\"practical applications\"\n\n\n }\n\n\n)\n\n\n\n\nPrompt Engineering Studio\nLearn more about Portkey’s prompt management features\n\n\n\n4. Guardrails for Safe Crews\n\n\nGuardrails ensure your CrewAI agents operate safely and respond appropriately in all situations.\n\n\nWhy Use Guardrails?\n\n\nCrewAI agents can experience various failure modes:\n\n\n\n\nGenerating harmful or inappropriate content\n\n\nLeaking sensitive information like PII\n\n\nHallucinating incorrect information\n\n\nGenerating outputs in incorrect formats\n\n\n\n\nPortkey’s guardrails add protections for both inputs and outputs.\n\n\nImplementing Guardrails\n\n\nCopy\nAsk AI\nfrom\n crewai \nimport\n Agent, \nLLM\n\n\nfrom\n portkey_ai \nimport\n createHeaders, \nPORTKEY_GATEWAY_URL",
646 "metadata": {
647 "source": "https://docs.crewai.com/en/observability/portkey",
648 "title": "Portkey Integration - CrewAI"
649 }
650 },
651 {
652 "id": "9a90ceb0-8880-43de-b1a1-44aca6c21b06",
653 "source": "https://docs.crewai.com/en/observability/portkey",
654 "title": "Portkey Integration - CrewAI",
655 "chunk_index": 18,
656 "text": "Implementing Guardrails\n\n\nCopy\nAsk AI\nfrom\n crewai \nimport\n Agent, \nLLM\n\n\nfrom\n portkey_ai \nimport\n createHeaders, \nPORTKEY_GATEWAY_URL\n\n\n\n\n# Create LLM with guardrails\n\n\nportkey_llm \n=\n LLM(\n\n\n model\n=\n\"gpt-4o\"\n,\n\n\n base_url\n=\nPORTKEY_GATEWAY_URL\n,\n\n\n api_key\n=\n\"dummy\"\n,\n\n\n extra_headers\n=\ncreateHeaders(\n\n\n api_key\n=\n\"YOUR_PORTKEY_API_KEY\"\n,\n\n\n virtual_key\n=\n\"YOUR_OPENAI_VIRTUAL_KEY\"\n,\n\n\n config\n=\n{\n\n\n \"input_guardrails\"\n: [\n\"guardrails-id-xxx\"\n, \n\"guardrails-id-yyy\"\n],\n\n\n \"output_guardrails\"\n: [\n\"guardrails-id-zzz\"\n]\n\n\n }\n\n\n )\n\n\n)\n\n\n\n\n# Create agent with guardrailed LLM\n\n\nresearcher \n=\n Agent(\n\n\n role\n=\n\"Senior Research Scientist\"\n,\n\n\n goal\n=\n\"Discover groundbreaking insights about the assigned topic\"\n,\n\n\n backstory\n=\n\"You are an expert researcher with deep domain knowledge.\"\n,\n\n\n verbose\n=\nTrue\n,\n\n\n llm\n=\nportkey_llm\n\n\n)\n\n\n\n\nPortkey’s guardrails can:",
657 "metadata": {
658 "source": "https://docs.crewai.com/en/observability/portkey",
659 "title": "Portkey Integration - CrewAI"
660 }
661 },
662 {
663 "id": "c8522638-2168-40a4-8c64-9aa0ece14d36",
664 "source": "https://docs.crewai.com/en/observability/portkey",
665 "title": "Portkey Integration - CrewAI",
666 "chunk_index": 19,
667 "text": "backstory\n=\n\"You are an expert researcher with deep domain knowledge.\"\n,\n\n\n verbose\n=\nTrue\n,\n\n\n llm\n=\nportkey_llm\n\n\n)\n\n\n\n\nPortkey’s guardrails can:\n\n\n\n\nDetect and redact PII in both inputs and outputs\n\n\nFilter harmful or inappropriate content\n\n\nValidate response formats against schemas\n\n\nCheck for hallucinations against ground truth\n\n\nApply custom business logic and rules\n\n\n\n\nLearn More About Guardrails\nExplore Portkey’s guardrail features to enhance agent safety\n\n\n\n5. User Tracking with Metadata\n\n\nTrack individual users through your CrewAI agents using Portkey’s metadata system.\n\n\nWhat is Metadata in Portkey?\n\n\nMetadata allows you to associate custom data with each request, enabling filtering, segmentation, and analytics. The special \n_user\n field is specifically designed for user tracking.\n\n\nCopy\nAsk AI\nfrom\n crewai \nimport\n Agent, \nLLM\n\n\nfrom\n portkey_ai \nimport\n createHeaders, \nPORTKEY_GATEWAY_URL\n\n\n\n\n# Configure LLM with user tracking\n\n\nportkey_llm \n=\n LLM(",
668 "metadata": {
669 "source": "https://docs.crewai.com/en/observability/portkey",
670 "title": "Portkey Integration - CrewAI"
671 }
672 },
673 {
674 "id": "590cd0ce-7a07-40f6-a994-b33dc25fdc00",
675 "source": "https://docs.crewai.com/en/observability/portkey",
676 "title": "Portkey Integration - CrewAI",
677 "chunk_index": 20,
678 "text": "Copy\nAsk AI\nfrom\n crewai \nimport\n Agent, \nLLM\n\n\nfrom\n portkey_ai \nimport\n createHeaders, \nPORTKEY_GATEWAY_URL\n\n\n\n\n# Configure LLM with user tracking\n\n\nportkey_llm \n=\n LLM(\n\n\n model\n=\n\"gpt-4o\"\n,\n\n\n base_url\n=\nPORTKEY_GATEWAY_URL\n,\n\n\n api_key\n=\n\"dummy\"\n,\n\n\n extra_headers\n=\ncreateHeaders(\n\n\n api_key\n=\n\"YOUR_PORTKEY_API_KEY\"\n,\n\n\n virtual_key\n=\n\"YOUR_OPENAI_VIRTUAL_KEY\"\n,\n\n\n metadata\n=\n{\n\n\n \"_user\"\n: \n\"user_123\"\n, \n# Special _user field for user analytics\n\n\n \"user_tier\"\n: \n\"premium\"\n,\n\n\n \"user_company\"\n: \n\"Acme Corp\"\n,\n\n\n \"session_id\"\n: \n\"abc-123\"\n\n\n }\n\n\n )\n\n\n)\n\n\n\n\n# Create agent with tracked LLM\n\n\nresearcher \n=\n Agent(\n\n\n role\n=\n\"Senior Research Scientist\"\n,\n\n\n goal\n=\n\"Discover groundbreaking insights about the assigned topic\"\n,\n\n\n backstory\n=\n\"You are an expert researcher with deep domain knowledge.\"\n,\n\n\n verbose\n=\nTrue\n,\n\n\n llm\n=\nportkey_llm\n\n\n)\n\n\n\n\nFilter Analytics by User",
679 "metadata": {
680 "source": "https://docs.crewai.com/en/observability/portkey",
681 "title": "Portkey Integration - CrewAI"
682 }
683 },
684 {
685 "id": "7c4b3170-83dc-4679-ba50-95fa59b15ad5",
686 "source": "https://docs.crewai.com/en/observability/portkey",
687 "title": "Portkey Integration - CrewAI",
688 "chunk_index": 21,
689 "text": "backstory\n=\n\"You are an expert researcher with deep domain knowledge.\"\n,\n\n\n verbose\n=\nTrue\n,\n\n\n llm\n=\nportkey_llm\n\n\n)\n\n\n\n\nFilter Analytics by User\n\n\nWith metadata in place, you can filter analytics by user and analyze performance metrics on a per-user basis:\n\n\nFilter analytics by user\n\n\nThis enables:\n\n\n\n\nPer-user cost tracking and budgeting\n\n\nPersonalized user analytics\n\n\nTeam or organization-level metrics\n\n\nEnvironment-specific monitoring (staging vs. production)\n\n\n\n\nLearn More About Metadata\nExplore how to use custom metadata to enhance your analytics\n\n\n\n6. Caching for Efficient Crews\n\n\nImplement caching to make your CrewAI agents more efficient and cost-effective:\n\n\nSimple Caching\nSemantic Caching\nCopy\nAsk AI\nfrom\n crewai \nimport\n Agent, \nLLM\n\n\nfrom\n portkey_ai \nimport\n createHeaders, \nPORTKEY_GATEWAY_URL\n\n\n\n\n# Configure LLM with simple caching\n\n\nportkey_llm \n=\n LLM(\n\n\n model\n=\n\"gpt-4o\"\n,\n\n\n base_url\n=\nPORTKEY_GATEWAY_URL\n,\n\n\n api_key\n=\n\"dummy\"\n,",
690 "metadata": {
691 "source": "https://docs.crewai.com/en/observability/portkey",
692 "title": "Portkey Integration - CrewAI"
693 }
694 },
695 {
696 "id": "6d84ce04-25fd-400e-946d-3f74d26db690",
697 "source": "https://docs.crewai.com/en/observability/portkey",
698 "title": "Portkey Integration - CrewAI",
699 "chunk_index": 22,
700 "text": "# Configure LLM with simple caching\n\n\nportkey_llm \n=\n LLM(\n\n\n model\n=\n\"gpt-4o\"\n,\n\n\n base_url\n=\nPORTKEY_GATEWAY_URL\n,\n\n\n api_key\n=\n\"dummy\"\n,\n\n\n extra_headers\n=\ncreateHeaders(\n\n\n api_key\n=\n\"YOUR_PORTKEY_API_KEY\"\n,\n\n\n virtual_key\n=\n\"YOUR_OPENAI_VIRTUAL_KEY\"\n,\n\n\n config\n=\n{\n\n\n \"cache\"\n: {\n\n\n \"mode\"\n: \n\"simple\"\n\n\n }\n\n\n }\n\n\n )\n\n\n)\n\n\n\n\n# Create agent with cached LLM\n\n\nresearcher \n=\n Agent(\n\n\n role\n=\n\"Senior Research Scientist\"\n,\n\n\n goal\n=\n\"Discover groundbreaking insights about the assigned topic\"\n,\n\n\n backstory\n=\n\"You are an expert researcher with deep domain knowledge.\"\n,\n\n\n verbose\n=\nTrue\n,\n\n\n llm\n=\nportkey_llm\n\n\n)\n\n\nSimple caching performs exact matches on input prompts, caching identical requests to avoid redundant model executions.\nCopy\nAsk AI\nfrom\n crewai \nimport\n Agent, \nLLM\n\n\nfrom\n portkey_ai \nimport\n createHeaders, \nPORTKEY_GATEWAY_URL\n\n\n\n\n# Configure LLM with simple caching",
701 "metadata": {
702 "source": "https://docs.crewai.com/en/observability/portkey",
703 "title": "Portkey Integration - CrewAI"
704 }
705 },
706 {
707 "id": "fda9b67c-9c9b-4157-8538-b42ff5a25524",
708 "source": "https://docs.crewai.com/en/observability/portkey",
709 "title": "Portkey Integration - CrewAI",
710 "chunk_index": 23,
711 "text": "from\n portkey_ai \nimport\n createHeaders, \nPORTKEY_GATEWAY_URL\n\n\n\n\n# Configure LLM with simple caching\n\n\nportkey_llm \n=\n LLM(\n\n\n model\n=\n\"gpt-4o\"\n,\n\n\n base_url\n=\nPORTKEY_GATEWAY_URL\n,\n\n\n api_key\n=\n\"dummy\"\n,\n\n\n extra_headers\n=\ncreateHeaders(\n\n\n api_key\n=\n\"YOUR_PORTKEY_API_KEY\"\n,\n\n\n virtual_key\n=\n\"YOUR_OPENAI_VIRTUAL_KEY\"\n,\n\n\n config\n=\n{\n\n\n \"cache\"\n: {\n\n\n \"mode\"\n: \n\"simple\"\n\n\n }\n\n\n }\n\n\n )\n\n\n)\n\n\n\n\n# Create agent with cached LLM\n\n\nresearcher \n=\n Agent(\n\n\n role\n=\n\"Senior Research Scientist\"\n,\n\n\n goal\n=\n\"Discover groundbreaking insights about the assigned topic\"\n,\n\n\n backstory\n=\n\"You are an expert researcher with deep domain knowledge.\"\n,\n\n\n verbose\n=\nTrue\n,\n\n\n llm\n=\nportkey_llm\n\n\n)\n\n\nSimple caching performs exact matches on input prompts, caching identical requests to avoid redundant model executions.\nCopy\nAsk AI\nfrom\n crewai \nimport\n Agent, \nLLM",
712 "metadata": {
713 "source": "https://docs.crewai.com/en/observability/portkey",
714 "title": "Portkey Integration - CrewAI"
715 }
716 },
717 {
718 "id": "d636810a-6f5f-43ee-aca1-e6609ebacc2b",
719 "source": "https://docs.crewai.com/en/observability/portkey",
720 "title": "Portkey Integration - CrewAI",
721 "chunk_index": 24,
722 "text": "llm\n=\nportkey_llm\n\n\n)\n\n\nSimple caching performs exact matches on input prompts, caching identical requests to avoid redundant model executions.\nCopy\nAsk AI\nfrom\n crewai \nimport\n Agent, \nLLM\n\n\nfrom\n portkey_ai \nimport\n createHeaders, \nPORTKEY_GATEWAY_URL\n\n\n\n\n# Configure LLM with semantic caching\n\n\nportkey_llm \n=\n LLM(\n\n\n model\n=\n\"gpt-4o\"\n,\n\n\n base_url\n=\nPORTKEY_GATEWAY_URL\n,\n\n\n api_key\n=\n\"dummy\"\n,\n\n\n extra_headers\n=\ncreateHeaders(\n\n\n api_key\n=\n\"YOUR_PORTKEY_API_KEY\"\n,\n\n\n virtual_key\n=\n\"YOUR_OPENAI_VIRTUAL_KEY\"\n,\n\n\n config\n=\n{\n\n\n \"cache\"\n: {\n\n\n \"mode\"\n: \n\"semantic\"\n\n\n }\n\n\n }\n\n\n )\n\n\n)\n\n\n\n\n# Create agent with semantically cached LLM\n\n\nresearcher \n=\n Agent(\n\n\n role\n=\n\"Senior Research Scientist\"\n,\n\n\n goal\n=\n\"Discover groundbreaking insights about the assigned topic\"\n,\n\n\n backstory\n=\n\"You are an expert researcher with deep domain knowledge.\"\n,\n\n\n verbose\n=\nTrue\n,\n\n\n llm\n=\nportkey_llm\n\n\n)",
723 "metadata": {
724 "source": "https://docs.crewai.com/en/observability/portkey",
725 "title": "Portkey Integration - CrewAI"
726 }
727 },
728 {
729 "id": "4a158c08-ed90-4f9d-a9b9-e85f127ed546",
730 "source": "https://docs.crewai.com/en/observability/portkey",
731 "title": "Portkey Integration - CrewAI",
732 "chunk_index": 25,
733 "text": "backstory\n=\n\"You are an expert researcher with deep domain knowledge.\"\n,\n\n\n verbose\n=\nTrue\n,\n\n\n llm\n=\nportkey_llm\n\n\n)\n\n\nSemantic caching considers the contextual similarity between input requests, caching responses for semantically similar inputs.\n\n\n\n7. Model Interoperability\n\n\nCrewAI supports multiple LLM providers, and Portkey extends this capability by providing access to over 200 LLMs through a unified interface. You can easily switch between different models without changing your core agent logic:\n\n\nCopy\nAsk AI\nfrom\n crewai \nimport\n Agent, \nLLM\n\n\nfrom\n portkey_ai \nimport\n createHeaders, \nPORTKEY_GATEWAY_URL\n\n\n\n\n# Set up LLMs with different providers\n\n\nopenai_llm \n=\n LLM(\n\n\n model\n=\n\"gpt-4o\"\n,\n\n\n base_url\n=\nPORTKEY_GATEWAY_URL\n,\n\n\n api_key\n=\n\"dummy\"\n,\n\n\n extra_headers\n=\ncreateHeaders(\n\n\n api_key\n=\n\"YOUR_PORTKEY_API_KEY\"\n,\n\n\n virtual_key\n=\n\"YOUR_OPENAI_VIRTUAL_KEY\"\n\n\n )\n\n\n)\n\n\n\n\nanthropic_llm \n=\n LLM(",
734 "metadata": {
735 "source": "https://docs.crewai.com/en/observability/portkey",
736 "title": "Portkey Integration - CrewAI"
737 }
738 },
739 {
740 "id": "66aa9473-6fef-4205-b1c5-4b22678c84ea",
741 "source": "https://docs.crewai.com/en/observability/portkey",
742 "title": "Portkey Integration - CrewAI",
743 "chunk_index": 26,
744 "text": "api_key\n=\n\"dummy\"\n,\n\n\n extra_headers\n=\ncreateHeaders(\n\n\n api_key\n=\n\"YOUR_PORTKEY_API_KEY\"\n,\n\n\n virtual_key\n=\n\"YOUR_OPENAI_VIRTUAL_KEY\"\n\n\n )\n\n\n)\n\n\n\n\nanthropic_llm \n=\n LLM(\n\n\n model\n=\n\"claude-3-5-sonnet-latest\"\n,\n\n\n max_tokens\n=\n1000\n,\n\n\n base_url\n=\nPORTKEY_GATEWAY_URL\n,\n\n\n api_key\n=\n\"dummy\"\n,\n\n\n extra_headers\n=\ncreateHeaders(\n\n\n api_key\n=\n\"YOUR_PORTKEY_API_KEY\"\n,\n\n\n virtual_key\n=\n\"YOUR_ANTHROPIC_VIRTUAL_KEY\"\n\n\n )\n\n\n)\n\n\n\n\n# Choose which LLM to use for each agent based on your needs\n\n\nresearcher \n=\n Agent(\n\n\n role\n=\n\"Senior Research Scientist\"\n,\n\n\n goal\n=\n\"Discover groundbreaking insights about the assigned topic\"\n,\n\n\n backstory\n=\n\"You are an expert researcher with deep domain knowledge.\"\n,\n\n\n verbose\n=\nTrue\n,\n\n\n llm\n=\nopenai_llm \n# Use anthropic_llm for Anthropic\n\n\n)\n\n\n\n\nPortkey provides access to LLMs from providers including:\n\n\n\n\nOpenAI (GPT-4o, GPT-4 Turbo, etc.)",
745 "metadata": {
746 "source": "https://docs.crewai.com/en/observability/portkey",
747 "title": "Portkey Integration - CrewAI"
748 }
749 },
750 {
751 "id": "ca2bcbcb-2cb7-42bc-86b7-59d5517cd70d",
752 "source": "https://docs.crewai.com/en/observability/portkey",
753 "title": "Portkey Integration - CrewAI",
754 "chunk_index": 27,
755 "text": "verbose\n=\nTrue\n,\n\n\n llm\n=\nopenai_llm \n# Use anthropic_llm for Anthropic\n\n\n)\n\n\n\n\nPortkey provides access to LLMs from providers including:\n\n\n\n\nOpenAI (GPT-4o, GPT-4 Turbo, etc.)\n\n\nAnthropic (Claude 3.5 Sonnet, Claude 3 Opus, etc.)\n\n\nMistral AI (Mistral Large, Mistral Medium, etc.)\n\n\nGoogle Vertex AI (Gemini 1.5 Pro, etc.)\n\n\nCohere (Command, Command-R, etc.)\n\n\nAWS Bedrock (Claude, Titan, etc.)\n\n\nLocal/Private Models\n\n\n\n\nSupported Providers\nSee the full list of LLM providers supported by Portkey\n\n\n\nSet Up Enterprise Governance for CrewAI\n\n\nWhy Enterprise Governance?\n\nIf you are using CrewAI inside your organization, you need to consider several governance aspects:\n\n\n\n\nCost Management\n: Controlling and tracking AI spending across teams\n\n\nAccess Control\n: Managing which teams can use specific models\n\n\nUsage Analytics\n: Understanding how AI is being used across the organization\n\n\nSecurity & Compliance\n: Maintaining enterprise security standards",
756 "metadata": {
757 "source": "https://docs.crewai.com/en/observability/portkey",
758 "title": "Portkey Integration - CrewAI"
759 }
760 },
761 {
762 "id": "3edc0c21-7a86-4dc6-93f9-73f8bd9ac89b",
763 "source": "https://docs.crewai.com/en/observability/portkey",
764 "title": "Portkey Integration - CrewAI",
765 "chunk_index": 28,
766 "text": "Usage Analytics\n: Understanding how AI is being used across the organization\n\n\nSecurity & Compliance\n: Maintaining enterprise security standards\n\n\nReliability\n: Ensuring consistent service across all users\n\n\n\n\nPortkey adds a comprehensive governance layer to address these enterprise needs. Let’s implement these controls step by step.\n\n\n1\nCreate Virtual Key\nVirtual Keys are Portkey’s secure way to manage your LLM provider API keys. They provide essential controls like:\n\n\nBudget limits for API usage\n\n\nRate limiting capabilities\n\n\nSecure API key storage\n\n\nTo create a virtual key:\nGo to \nVirtual Keys\n in the Portkey App. Save and copy the virtual key ID\nSave your virtual key ID - you’ll need it for the next step.\n2\nCreate Default Config\nConfigs in Portkey define how your requests are routed, with features like advanced routing, fallbacks, and retries.\nTo create your config:\n\n\nGo to \nConfigs\n in Portkey dashboard\n\n\nCreate new config with:\n\n\nCopy\nAsk AI\n{",
767 "metadata": {
768 "source": "https://docs.crewai.com/en/observability/portkey",
769 "title": "Portkey Integration - CrewAI"
770 }
771 },
772 {
773 "id": "6e9663fa-dbe9-4c95-838b-dc03ec603187",
774 "source": "https://docs.crewai.com/en/observability/portkey",
775 "title": "Portkey Integration - CrewAI",
776 "chunk_index": 29,
777 "text": "Go to \nConfigs\n in Portkey dashboard\n\n\nCreate new config with:\n\n\nCopy\nAsk AI\n{\n\n\n \"virtual_key\"\n: \n\"YOUR_VIRTUAL_KEY_FROM_STEP1\"\n,\n\n\n \t\"override_params\"\n: {\n\n\n \"model\"\n: \n\"gpt-4o\"\n // Your preferred model name\n\n\n }\n\n\n}\n\n\n\n\n\n\nSave and note the Config name for the next step\n\n\n3\nConfigure Portkey API Key\nNow create a Portkey API key and attach the config you created in Step 2:\n\n\nGo to \nAPI Keys\n in Portkey and Create new API key\n\n\nSelect your config from \nStep 2\n\n\nGenerate and save your API key\n\n\n4\nConnect to CrewAI\nAfter setting up your Portkey API key with the attached config, connect it to your CrewAI agents:\nCopy\nAsk AI\nfrom\n crewai \nimport\n Agent, \nLLM\n\n\nfrom\n portkey_ai \nimport\n PORTKEY_GATEWAY_URL\n\n\n\n\n# Configure LLM with your API key\n\n\nportkey_llm \n=\n LLM(\n\n\n model\n=\n\"gpt-4o\"\n,\n\n\n base_url\n=\nPORTKEY_GATEWAY_URL\n,\n\n\n api_key\n=\n\"YOUR_PORTKEY_API_KEY\"\n\n\n)\n\n\n\n\n# Create agent with Portkey-enabled LLM\n\n\nresearcher \n=\n Agent(",
778 "metadata": {
779 "source": "https://docs.crewai.com/en/observability/portkey",
780 "title": "Portkey Integration - CrewAI"
781 }
782 },
783 {
784 "id": "aacff62d-a30d-41ee-82b4-017a9d33a52f",
785 "source": "https://docs.crewai.com/en/observability/portkey",
786 "title": "Portkey Integration - CrewAI",
787 "chunk_index": 30,
788 "text": "portkey_llm \n=\n LLM(\n\n\n model\n=\n\"gpt-4o\"\n,\n\n\n base_url\n=\nPORTKEY_GATEWAY_URL\n,\n\n\n api_key\n=\n\"YOUR_PORTKEY_API_KEY\"\n\n\n)\n\n\n\n\n# Create agent with Portkey-enabled LLM\n\n\nresearcher \n=\n Agent(\n\n\n role\n=\n\"Senior Research Scientist\"\n,\n\n\n goal\n=\n\"Discover groundbreaking insights about the assigned topic\"\n,\n\n\n backstory\n=\n\"You are an expert researcher with deep domain knowledge.\"\n,\n\n\n verbose\n=\nTrue\n,\n\n\n llm\n=\nportkey_llm\n\n\n)\n\n\n\n\nStep 1: Implement Budget Controls & Rate Limits\n\nStep 1: Implement Budget Controls & Rate Limits\nVirtual Keys enable granular control over LLM access at the team/department level. This helps you:\n\n\nSet up \nbudget limits\n\n\nPrevent unexpected usage spikes using Rate limits\n\n\nTrack departmental spending\n\n\n\nSetting Up Department-Specific Controls:\n\n\nNavigate to \nVirtual Keys\n in Portkey dashboard\n\n\nCreate new Virtual Key for each department with budget limits and rate limits\n\n\nConfigure department-specific limits",
789 "metadata": {
790 "source": "https://docs.crewai.com/en/observability/portkey",
791 "title": "Portkey Integration - CrewAI"
792 }
793 },
794 {
795 "id": "c467048a-4be0-4bba-9927-755df3fa1f01",
796 "source": "https://docs.crewai.com/en/observability/portkey",
797 "title": "Portkey Integration - CrewAI",
798 "chunk_index": 31,
799 "text": "Navigate to \nVirtual Keys\n in Portkey dashboard\n\n\nCreate new Virtual Key for each department with budget limits and rate limits\n\n\nConfigure department-specific limits\n\n\nStep 2: Define Model Access Rules\n\nStep 2: Define Model Access Rules\nAs your AI usage scales, controlling which teams can access specific models becomes crucial. Portkey Configs provide this control layer with features like:\n\nAccess Control Features:\n\n\nModel Restrictions\n: Limit access to specific models\n\n\nData Protection\n: Implement guardrails for sensitive data\n\n\nReliability Controls\n: Add fallbacks and retry logic\n\n\n\nExample Configuration:\nHere’s a basic configuration to route requests to OpenAI, specifically using GPT-4o:\nCopy\nAsk AI\n{\n\n\n\t\"strategy\"\n: {\n\n\n\t\t\"mode\"\n: \n\"single\"\n\n\n\t},\n\n\n\t\"targets\"\n: [\n\n\n\t\t{\n\n\n\t\t\t\"virtual_key\"\n: \n\"YOUR_OPENAI_VIRTUAL_KEY\"\n,\n\n\n\t\t\t\"override_params\"\n: {\n\n\n\t\t\t\t\"model\"\n: \n\"gpt-4o\"\n\n\n\t\t\t}\n\n\n\t\t}\n\n\n\t]\n\n\n}",
800 "metadata": {
801 "source": "https://docs.crewai.com/en/observability/portkey",
802 "title": "Portkey Integration - CrewAI"
803 }
804 },
805 {
806 "id": "de69fb15-2a68-4ce3-9af5-aa0a2c52bc16",
807 "source": "https://docs.crewai.com/en/observability/portkey",
808 "title": "Portkey Integration - CrewAI",
809 "chunk_index": 32,
810 "text": "\"strategy\"\n: {\n\n\n\t\t\"mode\"\n: \n\"single\"\n\n\n\t},\n\n\n\t\"targets\"\n: [\n\n\n\t\t{\n\n\n\t\t\t\"virtual_key\"\n: \n\"YOUR_OPENAI_VIRTUAL_KEY\"\n,\n\n\n\t\t\t\"override_params\"\n: {\n\n\n\t\t\t\t\"model\"\n: \n\"gpt-4o\"\n\n\n\t\t\t}\n\n\n\t\t}\n\n\n\t]\n\n\n}\n\n\nCreate your config on the \nConfigs page\n in your Portkey dashboard.\nConfigs can be updated anytime to adjust controls without affecting running applications.\nStep 3: Implement Access Controls\n\nStep 3: Implement Access Controls\nCreate User-specific API keys that automatically:\n\n\nTrack usage per user/team with the help of virtual keys\n\n\nApply appropriate configs to route requests\n\n\nCollect relevant metadata to filter logs\n\n\nEnforce access permissions\n\n\nCreate API keys through:\n\n\nPortkey App\n\n\nAPI Key Management API\n\n\nExample using Python SDK:\nCopy\nAsk AI\nfrom\n portkey_ai \nimport\n Portkey\n\n\n\n\nportkey \n=\n Portkey(\napi_key\n=\n\"YOUR_ADMIN_API_KEY\"\n)\n\n\n\n\napi_key \n=\n portkey.api_keys.create(\n\n\n name\n=\n\"engineering-team\"\n,\n\n\n type\n=\n\"organisation\"\n,\n\n\n workspace_id\n=\n\"YOUR_WORKSPACE_ID\"\n,",
811 "metadata": {
812 "source": "https://docs.crewai.com/en/observability/portkey",
813 "title": "Portkey Integration - CrewAI"
814 }
815 },
816 {
817 "id": "af599c1d-714b-4e62-818c-e16ee191ba3e",
818 "source": "https://docs.crewai.com/en/observability/portkey",
819 "title": "Portkey Integration - CrewAI",
820 "chunk_index": 33,
821 "text": "api_key \n=\n portkey.api_keys.create(\n\n\n name\n=\n\"engineering-team\"\n,\n\n\n type\n=\n\"organisation\"\n,\n\n\n workspace_id\n=\n\"YOUR_WORKSPACE_ID\"\n,\n\n\n defaults\n=\n{\n\n\n \"config_id\"\n: \n\"your-config-id\"\n,\n\n\n \"metadata\"\n: {\n\n\n \"environment\"\n: \n\"production\"\n,\n\n\n \"department\"\n: \n\"engineering\"\n\n\n }\n\n\n },\n\n\n scopes\n=\n[\n\"logs.view\"\n, \n\"configs.read\"\n]\n\n\n)\n\n\nFor detailed key management instructions, see our \nAPI Keys documentation\n.\nStep 4: Deploy & Monitor\n\nStep 4: Deploy & Monitor\nAfter distributing API keys to your team members, your enterprise-ready CrewAI setup is ready to go. Each team member can now use their designated API keys with appropriate access levels and budget controls.\nMonitor usage in Portkey dashboard:\n\n\nCost tracking by department\n\n\nModel usage patterns\n\n\nRequest volumes\n\n\nError rates\n\n\n\n\n\nEnterprise Features Now Available\nYour CrewAI integration now has:\n\n\nDepartmental budget controls\n\n\nModel access governance",
822 "metadata": {
823 "source": "https://docs.crewai.com/en/observability/portkey",
824 "title": "Portkey Integration - CrewAI"
825 }
826 },
827 {
828 "id": "ed07075b-ab8a-45eb-8ede-27d82ba1503f",
829 "source": "https://docs.crewai.com/en/observability/portkey",
830 "title": "Portkey Integration - CrewAI",
831 "chunk_index": 34,
832 "text": "Model usage patterns\n\n\nRequest volumes\n\n\nError rates\n\n\n\n\n\nEnterprise Features Now Available\nYour CrewAI integration now has:\n\n\nDepartmental budget controls\n\n\nModel access governance\n\n\nUsage tracking & attribution\n\n\nSecurity guardrails\n\n\nReliability features\n\n\n\n\n\nFrequently Asked Questions",
833 "metadata": {
834 "source": "https://docs.crewai.com/en/observability/portkey",
835 "title": "Portkey Integration - CrewAI"
836 }
837 },
838 {
839 "id": "a07f5e45-ed55-4737-85c2-cb5f35362893",
840 "source": "https://docs.crewai.com/en/observability/portkey",
841 "title": "Portkey Integration - CrewAI",
842 "chunk_index": 35,
843 "text": "How does Portkey enhance CrewAI?\nPortkey adds production-readiness to CrewAI through comprehensive observability (traces, logs, metrics), reliability features (fallbacks, retries, caching), and access to 200+ LLMs through a unified interface. This makes it easier to debug, optimize, and scale your agent applications.\nCan I use Portkey with existing CrewAI applications?\nYes! Portkey integrates seamlessly with existing CrewAI applications. You just need to update your LLM configuration code with the Portkey-enabled version. The rest of your agent and crew code remains unchanged.\nDoes Portkey work with all CrewAI features?\nPortkey supports all CrewAI features, including agents, tools, human-in-the-loop workflows, and all task process types (sequential, hierarchical, etc.). It adds observability and reliability without limiting any of the framework’s functionality.\nCan I track usage across multiple agents in a crew?\nYes, Portkey allows you to use a consistent \ntrace_id",
844 "metadata": {
845 "source": "https://docs.crewai.com/en/observability/portkey",
846 "title": "Portkey Integration - CrewAI"
847 }
848 },
849 {
850 "id": "8cf76c8a-4ce6-4f5c-a245-8e03ef08475a",
851 "source": "https://docs.crewai.com/en/observability/portkey",
852 "title": "Portkey Integration - CrewAI",
853 "chunk_index": 36,
854 "text": "Can I track usage across multiple agents in a crew?\nYes, Portkey allows you to use a consistent \ntrace_id\n across multiple agents in a crew to track the entire workflow. This is especially useful for complex crews where you want to understand the full execution path across multiple agents.\nHow do I filter logs and traces for specific crew runs?\nPortkey allows you to add custom metadata to your LLM configuration, which you can then use for filtering. Add fields like \ncrew_name\n, \ncrew_type\n, or \nsession_id\n to easily find and analyze specific crew executions.\nCan I use my own API keys with Portkey?\nYes! Portkey uses your own API keys for the various LLM providers. It securely stores them as virtual keys, allowing you to easily manage and rotate keys without changing your code.",
855 "metadata": {
856 "source": "https://docs.crewai.com/en/observability/portkey",
857 "title": "Portkey Integration - CrewAI"
858 }
859 },
860 {
861 "id": "912faab8-7ea0-43f9-aa2a-30789341608b",
862 "source": "https://docs.crewai.com/en/observability/portkey",
863 "title": "Portkey Integration - CrewAI",
864 "chunk_index": 37,
865 "text": "\nResources\n\n\nCrewAI Docs\nOfficial CrewAI documentation\nBook a Demo\nGet personalized guidance on implementing this integration\nAssistant\nResponses are generated using AI and may contain mistakes.\nWas this page helpful?\nYes\nNo\nPatronus AI Evaluation\nWeave Integration\nwebsite\nx\ngithub\nlinkedin\nyoutube\nreddit\nPowered by Mintlify\nOn this page\nIntroduction\nInstallation & Setup\nProduction Features\n1. Enhanced Observability\n2. Reliability - Keep Your Crews Running Smoothly\n3. Prompting in CrewAI\n4. Guardrails for Safe Crews\n5. User Tracking with Metadata\n6. Caching for Efficient Crews\n7. Model Interoperability\nSet Up Enterprise Governance for CrewAI\nFrequently Asked Questions\nResources\nObservability\nPortkey Integration\nCopy page\nHow to use Portkey with CrewAI\n\n\n\nIntroduction\n\n\nPortkey enhances CrewAI with production-readiness features, turning your experimental agent crews into robust systems by providing:\n\n\n\n\nComplete observability\n of every agent step, tool use, and interaction",
866 "metadata": {
867 "source": "https://docs.crewai.com/en/observability/portkey",
868 "title": "Portkey Integration - CrewAI"
869 }
870 },
871 {
872 "id": "3f8d5576-5f41-467d-a95c-d5b732c15b33",
873 "source": "https://docs.crewai.com/en/observability/portkey",
874 "title": "Portkey Integration - CrewAI",
875 "chunk_index": 38,
876 "text": "Complete observability\n of every agent step, tool use, and interaction\n\n\nBuilt-in reliability\n with fallbacks, retries, and load balancing\n\n\nCost tracking and optimization\n to manage your AI spend\n\n\nAccess to 200+ LLMs\n through a single integration\n\n\nGuardrails\n to keep agent behavior safe and compliant\n\n\nVersion-controlled prompts\n for consistent agent performance\n\n\n\n\n\nInstallation & Setup\n\n\n1\nInstall the required packages\nCopy\nAsk AI\npip\n install\n -U\n crewai\n portkey-ai\n\n\nGenerate API Key\nCreate a Portkey API key with optional budget/rate limits from the \nPortkey dashboard\n. You can also attach configurations for reliability, caching, and more to this key. More on this later.\n3\nConfigure CrewAI with Portkey\nThe integration is simple - you just need to update the LLM configuration in your CrewAI setup:\nCopy\nAsk AI\nfrom\n crewai \nimport\n LLM\n\n\nfrom\n portkey_ai \nimport\n createHeaders, \nPORTKEY_GATEWAY_URL\n\n\n\n\n# Create an LLM instance with Portkey integration\n\n\ngpt_llm \n=\n LLM(",
877 "metadata": {
878 "source": "https://docs.crewai.com/en/observability/portkey",
879 "title": "Portkey Integration - CrewAI"
880 }
881 },
882 {
883 "id": "f13b6d9d-30e3-4475-bb9f-f04f57883c88",
884 "source": "https://docs.crewai.com/en/observability/portkey",
885 "title": "Portkey Integration - CrewAI",
886 "chunk_index": 39,
887 "text": "from\n portkey_ai \nimport\n createHeaders, \nPORTKEY_GATEWAY_URL\n\n\n\n\n# Create an LLM instance with Portkey integration\n\n\ngpt_llm \n=\n LLM(\n\n\n model\n=\n\"gpt-4o\"\n,\n\n\n base_url\n=\nPORTKEY_GATEWAY_URL\n,\n\n\n api_key\n=\n\"dummy\"\n, \n# We are using a Virtual key, so this is a placeholder\n\n\n extra_headers\n=\ncreateHeaders(\n\n\n api_key\n=\n\"YOUR_PORTKEY_API_KEY\"\n,\n\n\n virtual_key\n=\n\"YOUR_LLM_VIRTUAL_KEY\"\n,\n\n\n trace_id\n=\n\"unique-trace-id\"\n, \n# Optional, for request tracing\n\n\n )\n\n\n)\n\n\n\n\n#Use them in your Crew Agents like this:\n\n\n\n\n\t@agent\n\n\n\tdef\n lead_market_analyst\n(\nself\n) -> Agent:\n\n\n\t\treturn\n Agent(\n\n\n\t\t\tconfig\n=\nself\n.agents_config[\n'lead_market_analyst'\n],\n\n\n\t\t\tverbose\n=\nTrue\n,\n\n\n\t\t\tmemory\n=\nFalse\n,\n\n\n\t\t\tllm\n=\ngpt_llm\n\n\n\t\t)",
888 "metadata": {
889 "source": "https://docs.crewai.com/en/observability/portkey",
890 "title": "Portkey Integration - CrewAI"
891 }
892 },
893 {
894 "id": "3560edc4-f969-4341-91a7-b0066334486b",
895 "source": "https://docs.crewai.com/en/observability/portkey",
896 "title": "Portkey Integration - CrewAI",
897 "chunk_index": 40,
898 "text": "def\n lead_market_analyst\n(\nself\n) -> Agent:\n\n\n\t\treturn\n Agent(\n\n\n\t\t\tconfig\n=\nself\n.agents_config[\n'lead_market_analyst'\n],\n\n\n\t\t\tverbose\n=\nTrue\n,\n\n\n\t\t\tmemory\n=\nFalse\n,\n\n\n\t\t\tllm\n=\ngpt_llm\n\n\n\t\t)\n\n\n\n\nWhat are Virtual Keys?\n Virtual keys in Portkey securely store your LLM provider API keys (OpenAI, Anthropic, etc.) in an encrypted vault. They allow for easier key rotation and budget management. \nLearn more about virtual keys here\n.\n\n\n\nProduction Features\n\n\n\n1. Enhanced Observability\n\n\nPortkey provides comprehensive observability for your CrewAI agents, helping you understand exactly what’s happening during each execution.\n\n\nTraces\nLogs\nMetrics & Dashboards\nMetadata Filtering\nTraces provide a hierarchical view of your crew’s execution, showing the sequence of LLM calls, tool invocations, and state transitions.\nCopy\nAsk AI\n# Add trace_id to enable hierarchical tracing in Portkey\n\n\nportkey_llm \n=\n LLM(\n\n\n model\n=\n\"gpt-4o\"\n,\n\n\n base_url\n=\nPORTKEY_GATEWAY_URL\n,",
899 "metadata": {
900 "source": "https://docs.crewai.com/en/observability/portkey",
901 "title": "Portkey Integration - CrewAI"
902 }
903 },
904 {
905 "id": "28bcd08a-a20d-454f-b45a-a3ea338c15c7",
906 "source": "https://docs.crewai.com/en/observability/portkey",
907 "title": "Portkey Integration - CrewAI",
908 "chunk_index": 41,
909 "text": "portkey_llm \n=\n LLM(\n\n\n model\n=\n\"gpt-4o\"\n,\n\n\n base_url\n=\nPORTKEY_GATEWAY_URL\n,\n\n\n api_key\n=\n\"dummy\"\n,\n\n\n extra_headers\n=\ncreateHeaders(\n\n\n api_key\n=\n\"YOUR_PORTKEY_API_KEY\"\n,\n\n\n virtual_key\n=\n\"YOUR_OPENAI_VIRTUAL_KEY\"\n,\n\n\n trace_id\n=\n\"unique-session-id\"\n # Add unique trace ID\n\n\n )\n\n\n)\n\n\nTraces provide a hierarchical view of your crew’s execution, showing the sequence of LLM calls, tool invocations, and state transitions.\nCopy\nAsk AI\n# Add trace_id to enable hierarchical tracing in Portkey\n\n\nportkey_llm \n=\n LLM(\n\n\n model\n=\n\"gpt-4o\"\n,\n\n\n base_url\n=\nPORTKEY_GATEWAY_URL\n,\n\n\n api_key\n=\n\"dummy\"\n,\n\n\n extra_headers\n=\ncreateHeaders(\n\n\n api_key\n=\n\"YOUR_PORTKEY_API_KEY\"\n,\n\n\n virtual_key\n=\n\"YOUR_OPENAI_VIRTUAL_KEY\"\n,\n\n\n trace_id\n=\n\"unique-session-id\"\n # Add unique trace ID\n\n\n )\n\n\n)\n\n\nPortkey logs every interaction with LLMs, including:\n\n\nComplete request and response payloads\n\n\nLatency and token usage metrics",
910 "metadata": {
911 "source": "https://docs.crewai.com/en/observability/portkey",
912 "title": "Portkey Integration - CrewAI"
913 }
914 },
915 {
916 "id": "b0b0b2c2-2d71-414b-be1f-f2af273304f4",
917 "source": "https://docs.crewai.com/en/observability/portkey",
918 "title": "Portkey Integration - CrewAI",
919 "chunk_index": 42,
920 "text": ")\n\n\n)\n\n\nPortkey logs every interaction with LLMs, including:\n\n\nComplete request and response payloads\n\n\nLatency and token usage metrics\n\n\nCost calculations\n\n\nTool calls and function executions\n\n\nAll logs can be filtered by metadata, trace IDs, models, and more, making it easy to debug specific crew runs.\nPortkey provides built-in dashboards that help you:\n\n\nTrack cost and token usage across all crew runs\n\n\nAnalyze performance metrics like latency and success rates\n\n\nIdentify bottlenecks in your agent workflows\n\n\nCompare different crew configurations and LLMs\n\n\nYou can filter and segment all metrics by custom metadata to analyze specific crew types, user groups, or use cases.\nAdd custom metadata to your CrewAI LLM configuration to enable powerful filtering and segmentation:\nCopy\nAsk AI\nportkey_llm \n=\n LLM(\n\n\n model\n=\n\"gpt-4o\"\n,\n\n\n base_url\n=\nPORTKEY_GATEWAY_URL\n,\n\n\n api_key\n=\n\"dummy\"\n,\n\n\n extra_headers\n=\ncreateHeaders(\n\n\n api_key\n=\n\"YOUR_PORTKEY_API_KEY\"\n,",
921 "metadata": {
922 "source": "https://docs.crewai.com/en/observability/portkey",
923 "title": "Portkey Integration - CrewAI"
924 }
925 },
926 {
927 "id": "e581515d-33a3-483b-8ac1-9fac5ee32ad6",
928 "source": "https://docs.crewai.com/en/observability/portkey",
929 "title": "Portkey Integration - CrewAI",
930 "chunk_index": 43,
931 "text": "model\n=\n\"gpt-4o\"\n,\n\n\n base_url\n=\nPORTKEY_GATEWAY_URL\n,\n\n\n api_key\n=\n\"dummy\"\n,\n\n\n extra_headers\n=\ncreateHeaders(\n\n\n api_key\n=\n\"YOUR_PORTKEY_API_KEY\"\n,\n\n\n virtual_key\n=\n\"YOUR_OPENAI_VIRTUAL_KEY\"\n,\n\n\n metadata\n=\n{\n\n\n \"crew_type\"\n: \n\"research_crew\"\n,\n\n\n \"environment\"\n: \n\"production\"\n,\n\n\n \"_user\"\n: \n\"user_123\"\n, \n# Special _user field for user analytics\n\n\n \"request_source\"\n: \n\"mobile_app\"\n\n\n }\n\n\n )\n\n\n)\n\n\nThis metadata can be used to filter logs, traces, and metrics on the Portkey dashboard, allowing you to analyze specific crew runs, users, or environments.\n\n\n\n2. Reliability - Keep Your Crews Running Smoothly\n\n\nWhen running crews in production, things can go wrong - API rate limits, network issues, or provider outages. Portkey’s reliability features ensure your agents keep running smoothly even when problems occur.\n\n\nIt’s simple to enable fallback in your CrewAI setup by using a Portkey Config:",
932 "metadata": {
933 "source": "https://docs.crewai.com/en/observability/portkey",
934 "title": "Portkey Integration - CrewAI"
935 }
936 },
937 {
938 "id": "5d15025a-192e-4186-8af3-4a0c43ea3ba0",
939 "source": "https://docs.crewai.com/en/observability/portkey",
940 "title": "Portkey Integration - CrewAI",
941 "chunk_index": 44,
942 "text": "It’s simple to enable fallback in your CrewAI setup by using a Portkey Config:\n\n\nCopy\nAsk AI\nfrom\n crewai \nimport\n LLM\n\n\nfrom\n portkey_ai \nimport\n createHeaders, \nPORTKEY_GATEWAY_URL\n\n\n\n\n# Create LLM with fallback configuration\n\n\nportkey_llm \n=\n LLM(\n\n\n model\n=\n\"gpt-4o\"\n,\n\n\n max_tokens\n=\n1000\n,\n\n\n base_url\n=\nPORTKEY_GATEWAY_URL\n,\n\n\n api_key\n=\n\"dummy\"\n,\n\n\n extra_headers\n=\ncreateHeaders(\n\n\n api_key\n=\n\"YOUR_PORTKEY_API_KEY\"\n,\n\n\n config\n=\n{\n\n\n \"strategy\"\n: {\n\n\n \"mode\"\n: \n\"fallback\"\n\n\n },\n\n\n \"targets\"\n: [\n\n\n {\n\n\n \"provider\"\n: \n\"openai\"\n,\n\n\n \"api_key\"\n: \n\"YOUR_OPENAI_API_KEY\"\n,\n\n\n \"override_params\"\n: {\n\"model\"\n: \n\"gpt-4o\"\n}\n\n\n },\n\n\n {\n\n\n \"provider\"\n: \n\"anthropic\"\n,\n\n\n \"api_key\"\n: \n\"YOUR_ANTHROPIC_API_KEY\"\n,",
943 "metadata": {
944 "source": "https://docs.crewai.com/en/observability/portkey",
945 "title": "Portkey Integration - CrewAI"
946 }
947 },
948 {
949 "id": "b0a4f912-2041-4190-90c8-d06707c8bf62",
950 "source": "https://docs.crewai.com/en/observability/portkey",
951 "title": "Portkey Integration - CrewAI",
952 "chunk_index": 45,
953 "text": "},\n\n\n {\n\n\n \"provider\"\n: \n\"anthropic\"\n,\n\n\n \"api_key\"\n: \n\"YOUR_ANTHROPIC_API_KEY\"\n,\n\n\n \"override_params\"\n: {\n\"model\"\n: \n\"claude-3-opus-20240229\"\n}\n\n\n }\n\n\n ]\n\n\n }\n\n\n )\n\n\n)\n\n\n\n\n# Use this LLM configuration with your agents\n\n\n\n\nThis configuration will automatically try Claude if the GPT-4o request fails, ensuring your crew can continue operating.",
954 "metadata": {
955 "source": "https://docs.crewai.com/en/observability/portkey",
956 "title": "Portkey Integration - CrewAI"
957 }
958 },
959 {
960 "id": "459fa2c0-0b77-4d04-858c-281c0f4162c3",
961 "source": "https://docs.crewai.com/en/observability/portkey",
962 "title": "Portkey Integration - CrewAI",
963 "chunk_index": 46,
964 "text": "}\n\n\n )\n\n\n)\n\n\n\n\n# Use this LLM configuration with your agents\n\n\n\n\nThis configuration will automatically try Claude if the GPT-4o request fails, ensuring your crew can continue operating.\n\n\nAutomatic Retries\nHandles temporary failures automatically. If an LLM call fails, Portkey will retry the same request for the specified number of times - perfect for rate limits or network blips.\nRequest Timeouts\nPrevent your agents from hanging. Set timeouts to ensure you get responses (or can fail gracefully) within your required timeframes.\nConditional Routing\nSend different requests to different providers. Route complex reasoning to GPT-4, creative tasks to Claude, and quick responses to Gemini based on your needs.\nFallbacks\nKeep running even if your primary provider fails. Automatically switch to backup providers to maintain availability.\nLoad Balancing\nSpread requests across multiple API keys or providers. Great for high-volume crew operations and staying within rate limits.",
965 "metadata": {
966 "source": "https://docs.crewai.com/en/observability/portkey",
967 "title": "Portkey Integration - CrewAI"
968 }
969 },
970 {
971 "id": "1c81c032-e187-44ec-8d24-7a964181a6f8",
972 "source": "https://docs.crewai.com/en/observability/portkey",
973 "title": "Portkey Integration - CrewAI",
974 "chunk_index": 47,
975 "text": "\n3. Prompting in CrewAI\n\n\nPortkey’s Prompt Engineering Studio helps you create, manage, and optimize the prompts used in your CrewAI agents. Instead of hardcoding prompts or instructions, use Portkey’s prompt rendering API to dynamically fetch and apply your versioned prompts.\n\n\nManage prompts in Portkey's Prompt Library\n\n\nPrompt Playground\nUsing Prompt Templates\nPrompt Versioning\nMustache Templating for variables\nPrompt Playground is a place to compare, test and deploy perfect prompts for your AI application. It’s where you experiment with different models, test variables, compare outputs, and refine your prompt engineering strategy before deploying to production. It allows you to:\n\n\nIteratively develop prompts before using them in your agents\n\n\nTest prompts with different variables and models\n\n\nCompare outputs between different prompt versions\n\n\nCollaborate with team members on prompt development",
976 "metadata": {
977 "source": "https://docs.crewai.com/en/observability/portkey",
978 "title": "Portkey Integration - CrewAI"
979 }
980 },
981 {
982 "id": "788f5aab-9d20-4e91-a8af-77be52474887",
983 "source": "https://docs.crewai.com/en/observability/portkey",
984 "title": "Portkey Integration - CrewAI",
985 "chunk_index": 48,
986 "text": "Test prompts with different variables and models\n\n\nCompare outputs between different prompt versions\n\n\nCollaborate with team members on prompt development\n\n\nThis visual environment makes it easier to craft effective prompts for each step in your CrewAI agents’ workflow.\nPrompt Playground is a place to compare, test and deploy perfect prompts for your AI application. It’s where you experiment with different models, test variables, compare outputs, and refine your prompt engineering strategy before deploying to production. It allows you to:\n\n\nIteratively develop prompts before using them in your agents\n\n\nTest prompts with different variables and models\n\n\nCompare outputs between different prompt versions\n\n\nCollaborate with team members on prompt development",
987 "metadata": {
988 "source": "https://docs.crewai.com/en/observability/portkey",
989 "title": "Portkey Integration - CrewAI"
990 }
991 },
992 {
993 "id": "ec2f1436-dbcd-4021-a3a7-d0c7ad810f22",
994 "source": "https://docs.crewai.com/en/observability/portkey",
995 "title": "Portkey Integration - CrewAI",
996 "chunk_index": 49,
997 "text": "Test prompts with different variables and models\n\n\nCompare outputs between different prompt versions\n\n\nCollaborate with team members on prompt development\n\n\nThis visual environment makes it easier to craft effective prompts for each step in your CrewAI agents’ workflow.\nThe Prompt Render API retrieves your prompt templates with all parameters configured:\nCopy\nAsk AI\nfrom\n crewai \nimport\n Agent, \nLLM\n\n\nfrom\n portkey_ai \nimport\n createHeaders, \nPORTKEY_GATEWAY_URL\n, Portkey\n\n\n\n\n# Initialize Portkey admin client\n\n\nportkey_admin \n=\n Portkey(\napi_key\n=\n\"YOUR_PORTKEY_API_KEY\"\n)\n\n\n\n\n# Retrieve prompt using the render API\n\n\nprompt_data \n=\n portkey_client.prompts.render(\n\n\n prompt_id\n=\n\"YOUR_PROMPT_ID\"\n,\n\n\n variables\n=\n{\n\n\n \"agent_role\"\n: \n\"Senior Research Scientist\"\n,\n\n\n }\n\n\n)\n\n\n\n\nbackstory_agent_prompt\n=\nprompt_data.data.messages[\n0\n][\n\"content\"\n]\n\n\n\n\n\n\n# Set up LLM with Portkey integration\n\n\nportkey_llm \n=\n LLM(\n\n\n model\n=\n\"gpt-4o\"\n,",
998 "metadata": {
999 "source": "https://docs.crewai.com/en/observability/portkey",
1000 "title": "Portkey Integration - CrewAI"
1001 }
1002 },
1003 {
1004 "id": "172f012f-f20c-44a6-b7a9-d47b4a17b9ae",
1005 "source": "https://docs.crewai.com/en/observability/portkey",
1006 "title": "Portkey Integration - CrewAI",
1007 "chunk_index": 50,
1008 "text": "}\n\n\n)\n\n\n\n\nbackstory_agent_prompt\n=\nprompt_data.data.messages[\n0\n][\n\"content\"\n]\n\n\n\n\n\n\n# Set up LLM with Portkey integration\n\n\nportkey_llm \n=\n LLM(\n\n\n model\n=\n\"gpt-4o\"\n,\n\n\n base_url\n=\nPORTKEY_GATEWAY_URL\n,\n\n\n api_key\n=\n\"dummy\"\n,\n\n\n extra_headers\n=\ncreateHeaders(\n\n\n api_key\n=\n\"YOUR_PORTKEY_API_KEY\"\n,\n\n\n virtual_key\n=\n\"YOUR_OPENAI_VIRTUAL_KEY\"\n\n\n )\n\n\n)\n\n\n\n\n# Create agent using the rendered prompt\n\n\nresearcher \n=\n Agent(\n\n\n role\n=\n\"Senior Research Scientist\"\n,\n\n\n goal\n=\n\"Discover groundbreaking insights about the assigned topic\"\n,\n\n\n backstory\n=\nbackstory_agent, \n# Use the rendered prompt\n\n\n verbose\n=\nTrue\n,\n\n\n llm\n=\nportkey_llm\n\n\n)\n\n\nYou can:\n\n\nCreate multiple versions of the same prompt\n\n\nCompare performance between versions\n\n\nRoll back to previous versions if needed\n\n\nSpecify which version to use in your code:\n\n\nCopy\nAsk AI\n# Use a specific prompt version\n\n\nprompt_data \n=\n portkey_admin.prompts.render(",
1009 "metadata": {
1010 "source": "https://docs.crewai.com/en/observability/portkey",
1011 "title": "Portkey Integration - CrewAI"
1012 }
1013 },
1014 {
1015 "id": "f6ae0013-e002-4d56-80e9-a403700c074a",
1016 "source": "https://docs.crewai.com/en/observability/portkey",
1017 "title": "Portkey Integration - CrewAI",
1018 "chunk_index": 51,
1019 "text": "Roll back to previous versions if needed\n\n\nSpecify which version to use in your code:\n\n\nCopy\nAsk AI\n# Use a specific prompt version\n\n\nprompt_data \n=\n portkey_admin.prompts.render(\n\n\n prompt_id\n=\n\"YOUR_PROMPT_ID@version_number\"\n,\n\n\n variables\n=\n{\n\n\n \"agent_role\"\n: \n\"Senior Research Scientist\"\n,\n\n\n \"agent_goal\"\n: \n\"Discover groundbreaking insights\"\n\n\n }\n\n\n)\n\n\nPortkey prompts use Mustache-style templating for easy variable substitution:\nCopy\nAsk AI\nYou are a {{agent_role}} with expertise in {{domain}}.\n\n\n\n\nYour mission is to {{agent_goal}} by leveraging your knowledge\n\n\nand experience in the field.\n\n\n\n\nAlways maintain a {{tone}} tone and focus on providing {{focus_area}}.\n\n\nWhen rendering, simply pass the variables:\nCopy\nAsk AI\nprompt_data \n=\n portkey_admin.prompts.render(\n\n\n prompt_id\n=\n\"YOUR_PROMPT_ID\"\n,\n\n\n variables\n=\n{\n\n\n \"agent_role\"\n: \n\"Senior Research Scientist\"\n,\n\n\n \"domain\"\n: \n\"artificial intelligence\"\n,",
1020 "metadata": {
1021 "source": "https://docs.crewai.com/en/observability/portkey",
1022 "title": "Portkey Integration - CrewAI"
1023 }
1024 },
1025 {
1026 "id": "18bfd3a3-0324-4560-8780-0e768bf01527",
1027 "source": "https://docs.crewai.com/en/observability/portkey",
1028 "title": "Portkey Integration - CrewAI",
1029 "chunk_index": 52,
1030 "text": "prompt_id\n=\n\"YOUR_PROMPT_ID\"\n,\n\n\n variables\n=\n{\n\n\n \"agent_role\"\n: \n\"Senior Research Scientist\"\n,\n\n\n \"domain\"\n: \n\"artificial intelligence\"\n,\n\n\n \"agent_goal\"\n: \n\"discover groundbreaking insights\"\n,\n\n\n \"tone\"\n: \n\"professional\"\n,\n\n\n \"focus_area\"\n: \n\"practical applications\"\n\n\n }\n\n\n)\n\n\n\n\nPrompt Engineering Studio\nLearn more about Portkey’s prompt management features\n\n\n\n4. Guardrails for Safe Crews\n\n\nGuardrails ensure your CrewAI agents operate safely and respond appropriately in all situations.\n\n\nWhy Use Guardrails?\n\n\nCrewAI agents can experience various failure modes:\n\n\n\n\nGenerating harmful or inappropriate content\n\n\nLeaking sensitive information like PII\n\n\nHallucinating incorrect information\n\n\nGenerating outputs in incorrect formats\n\n\n\n\nPortkey’s guardrails add protections for both inputs and outputs.\n\n\nImplementing Guardrails\n\n\nCopy\nAsk AI\nfrom\n crewai \nimport\n Agent, \nLLM\n\n\nfrom\n portkey_ai \nimport\n createHeaders, \nPORTKEY_GATEWAY_URL",
1031 "metadata": {
1032 "source": "https://docs.crewai.com/en/observability/portkey",
1033 "title": "Portkey Integration - CrewAI"
1034 }
1035 },
1036 {
1037 "id": "210a1b2a-fa86-46bf-9398-12d00906fd5b",
1038 "source": "https://docs.crewai.com/en/observability/portkey",
1039 "title": "Portkey Integration - CrewAI",
1040 "chunk_index": 53,
1041 "text": "Implementing Guardrails\n\n\nCopy\nAsk AI\nfrom\n crewai \nimport\n Agent, \nLLM\n\n\nfrom\n portkey_ai \nimport\n createHeaders, \nPORTKEY_GATEWAY_URL\n\n\n\n\n# Create LLM with guardrails\n\n\nportkey_llm \n=\n LLM(\n\n\n model\n=\n\"gpt-4o\"\n,\n\n\n base_url\n=\nPORTKEY_GATEWAY_URL\n,\n\n\n api_key\n=\n\"dummy\"\n,\n\n\n extra_headers\n=\ncreateHeaders(\n\n\n api_key\n=\n\"YOUR_PORTKEY_API_KEY\"\n,\n\n\n virtual_key\n=\n\"YOUR_OPENAI_VIRTUAL_KEY\"\n,\n\n\n config\n=\n{\n\n\n \"input_guardrails\"\n: [\n\"guardrails-id-xxx\"\n, \n\"guardrails-id-yyy\"\n],\n\n\n \"output_guardrails\"\n: [\n\"guardrails-id-zzz\"\n]\n\n\n }\n\n\n )\n\n\n)\n\n\n\n\n# Create agent with guardrailed LLM\n\n\nresearcher \n=\n Agent(\n\n\n role\n=\n\"Senior Research Scientist\"\n,\n\n\n goal\n=\n\"Discover groundbreaking insights about the assigned topic\"\n,\n\n\n backstory\n=\n\"You are an expert researcher with deep domain knowledge.\"\n,\n\n\n verbose\n=\nTrue\n,\n\n\n llm\n=\nportkey_llm\n\n\n)\n\n\n\n\nPortkey’s guardrails can:",
1042 "metadata": {
1043 "source": "https://docs.crewai.com/en/observability/portkey",
1044 "title": "Portkey Integration - CrewAI"
1045 }
1046 },
1047 {
1048 "id": "3bca2f8f-daea-4ffb-9098-e65aad575d1d",
1049 "source": "https://docs.crewai.com/en/observability/portkey",
1050 "title": "Portkey Integration - CrewAI",
1051 "chunk_index": 54,
1052 "text": "backstory\n=\n\"You are an expert researcher with deep domain knowledge.\"\n,\n\n\n verbose\n=\nTrue\n,\n\n\n llm\n=\nportkey_llm\n\n\n)\n\n\n\n\nPortkey’s guardrails can:\n\n\n\n\nDetect and redact PII in both inputs and outputs\n\n\nFilter harmful or inappropriate content\n\n\nValidate response formats against schemas\n\n\nCheck for hallucinations against ground truth\n\n\nApply custom business logic and rules\n\n\n\n\nLearn More About Guardrails\nExplore Portkey’s guardrail features to enhance agent safety\n\n\n\n5. User Tracking with Metadata\n\n\nTrack individual users through your CrewAI agents using Portkey’s metadata system.\n\n\nWhat is Metadata in Portkey?\n\n\nMetadata allows you to associate custom data with each request, enabling filtering, segmentation, and analytics. The special \n_user\n field is specifically designed for user tracking.\n\n\nCopy\nAsk AI\nfrom\n crewai \nimport\n Agent, \nLLM\n\n\nfrom\n portkey_ai \nimport\n createHeaders, \nPORTKEY_GATEWAY_URL\n\n\n\n\n# Configure LLM with user tracking\n\n\nportkey_llm \n=\n LLM(",
1053 "metadata": {
1054 "source": "https://docs.crewai.com/en/observability/portkey",
1055 "title": "Portkey Integration - CrewAI"
1056 }
1057 },
1058 {
1059 "id": "c726c7f2-5f91-41db-be84-14436ad555d2",
1060 "source": "https://docs.crewai.com/en/observability/portkey",
1061 "title": "Portkey Integration - CrewAI",
1062 "chunk_index": 55,
1063 "text": "Copy\nAsk AI\nfrom\n crewai \nimport\n Agent, \nLLM\n\n\nfrom\n portkey_ai \nimport\n createHeaders, \nPORTKEY_GATEWAY_URL\n\n\n\n\n# Configure LLM with user tracking\n\n\nportkey_llm \n=\n LLM(\n\n\n model\n=\n\"gpt-4o\"\n,\n\n\n base_url\n=\nPORTKEY_GATEWAY_URL\n,\n\n\n api_key\n=\n\"dummy\"\n,\n\n\n extra_headers\n=\ncreateHeaders(\n\n\n api_key\n=\n\"YOUR_PORTKEY_API_KEY\"\n,\n\n\n virtual_key\n=\n\"YOUR_OPENAI_VIRTUAL_KEY\"\n,\n\n\n metadata\n=\n{\n\n\n \"_user\"\n: \n\"user_123\"\n, \n# Special _user field for user analytics\n\n\n \"user_tier\"\n: \n\"premium\"\n,\n\n\n \"user_company\"\n: \n\"Acme Corp\"\n,\n\n\n \"session_id\"\n: \n\"abc-123\"\n\n\n }\n\n\n )\n\n\n)\n\n\n\n\n# Create agent with tracked LLM\n\n\nresearcher \n=\n Agent(\n\n\n role\n=\n\"Senior Research Scientist\"\n,\n\n\n goal\n=\n\"Discover groundbreaking insights about the assigned topic\"\n,\n\n\n backstory\n=\n\"You are an expert researcher with deep domain knowledge.\"\n,\n\n\n verbose\n=\nTrue\n,\n\n\n llm\n=\nportkey_llm\n\n\n)\n\n\n\n\nFilter Analytics by User",
1064 "metadata": {
1065 "source": "https://docs.crewai.com/en/observability/portkey",
1066 "title": "Portkey Integration - CrewAI"
1067 }
1068 },
1069 {
1070 "id": "7c28bd51-3a21-48ec-92a4-e76dc29142df",
1071 "source": "https://docs.crewai.com/en/observability/portkey",
1072 "title": "Portkey Integration - CrewAI",
1073 "chunk_index": 56,
1074 "text": "backstory\n=\n\"You are an expert researcher with deep domain knowledge.\"\n,\n\n\n verbose\n=\nTrue\n,\n\n\n llm\n=\nportkey_llm\n\n\n)\n\n\n\n\nFilter Analytics by User\n\n\nWith metadata in place, you can filter analytics by user and analyze performance metrics on a per-user basis:\n\n\nFilter analytics by user\n\n\nThis enables:\n\n\n\n\nPer-user cost tracking and budgeting\n\n\nPersonalized user analytics\n\n\nTeam or organization-level metrics\n\n\nEnvironment-specific monitoring (staging vs. production)\n\n\n\n\nLearn More About Metadata\nExplore how to use custom metadata to enhance your analytics\n\n\n\n6. Caching for Efficient Crews\n\n\nImplement caching to make your CrewAI agents more efficient and cost-effective:\n\n\nSimple Caching\nSemantic Caching\nCopy\nAsk AI\nfrom\n crewai \nimport\n Agent, \nLLM\n\n\nfrom\n portkey_ai \nimport\n createHeaders, \nPORTKEY_GATEWAY_URL\n\n\n\n\n# Configure LLM with simple caching\n\n\nportkey_llm \n=\n LLM(\n\n\n model\n=\n\"gpt-4o\"\n,\n\n\n base_url\n=\nPORTKEY_GATEWAY_URL\n,\n\n\n api_key\n=\n\"dummy\"\n,",
1075 "metadata": {
1076 "source": "https://docs.crewai.com/en/observability/portkey",
1077 "title": "Portkey Integration - CrewAI"
1078 }
1079 },
1080 {
1081 "id": "83202c05-bbf1-44e0-865e-8ad5fb0f2e81",
1082 "source": "https://docs.crewai.com/en/observability/portkey",
1083 "title": "Portkey Integration - CrewAI",
1084 "chunk_index": 57,
1085 "text": "# Configure LLM with simple caching\n\n\nportkey_llm \n=\n LLM(\n\n\n model\n=\n\"gpt-4o\"\n,\n\n\n base_url\n=\nPORTKEY_GATEWAY_URL\n,\n\n\n api_key\n=\n\"dummy\"\n,\n\n\n extra_headers\n=\ncreateHeaders(\n\n\n api_key\n=\n\"YOUR_PORTKEY_API_KEY\"\n,\n\n\n virtual_key\n=\n\"YOUR_OPENAI_VIRTUAL_KEY\"\n,\n\n\n config\n=\n{\n\n\n \"cache\"\n: {\n\n\n \"mode\"\n: \n\"simple\"\n\n\n }\n\n\n }\n\n\n )\n\n\n)\n\n\n\n\n# Create agent with cached LLM\n\n\nresearcher \n=\n Agent(\n\n\n role\n=\n\"Senior Research Scientist\"\n,\n\n\n goal\n=\n\"Discover groundbreaking insights about the assigned topic\"\n,\n\n\n backstory\n=\n\"You are an expert researcher with deep domain knowledge.\"\n,\n\n\n verbose\n=\nTrue\n,\n\n\n llm\n=\nportkey_llm\n\n\n)\n\n\nSimple caching performs exact matches on input prompts, caching identical requests to avoid redundant model executions.\nCopy\nAsk AI\nfrom\n crewai \nimport\n Agent, \nLLM\n\n\nfrom\n portkey_ai \nimport\n createHeaders, \nPORTKEY_GATEWAY_URL\n\n\n\n\n# Configure LLM with simple caching",
1086 "metadata": {
1087 "source": "https://docs.crewai.com/en/observability/portkey",
1088 "title": "Portkey Integration - CrewAI"
1089 }
1090 },
1091 {
1092 "id": "4c890c5f-1f5a-4fbe-941a-e5839a87d21f",
1093 "source": "https://docs.crewai.com/en/observability/portkey",
1094 "title": "Portkey Integration - CrewAI",
1095 "chunk_index": 58,
1096 "text": "from\n portkey_ai \nimport\n createHeaders, \nPORTKEY_GATEWAY_URL\n\n\n\n\n# Configure LLM with simple caching\n\n\nportkey_llm \n=\n LLM(\n\n\n model\n=\n\"gpt-4o\"\n,\n\n\n base_url\n=\nPORTKEY_GATEWAY_URL\n,\n\n\n api_key\n=\n\"dummy\"\n,\n\n\n extra_headers\n=\ncreateHeaders(\n\n\n api_key\n=\n\"YOUR_PORTKEY_API_KEY\"\n,\n\n\n virtual_key\n=\n\"YOUR_OPENAI_VIRTUAL_KEY\"\n,\n\n\n config\n=\n{\n\n\n \"cache\"\n: {\n\n\n \"mode\"\n: \n\"simple\"\n\n\n }\n\n\n }\n\n\n )\n\n\n)\n\n\n\n\n# Create agent with cached LLM\n\n\nresearcher \n=\n Agent(\n\n\n role\n=\n\"Senior Research Scientist\"\n,\n\n\n goal\n=\n\"Discover groundbreaking insights about the assigned topic\"\n,\n\n\n backstory\n=\n\"You are an expert researcher with deep domain knowledge.\"\n,\n\n\n verbose\n=\nTrue\n,\n\n\n llm\n=\nportkey_llm\n\n\n)\n\n\nSimple caching performs exact matches on input prompts, caching identical requests to avoid redundant model executions.\nCopy\nAsk AI\nfrom\n crewai \nimport\n Agent, \nLLM",
1097 "metadata": {
1098 "source": "https://docs.crewai.com/en/observability/portkey",
1099 "title": "Portkey Integration - CrewAI"
1100 }
1101 },
1102 {
1103 "id": "9062f707-9786-4632-9ad0-c34171711921",
1104 "source": "https://docs.crewai.com/en/observability/portkey",
1105 "title": "Portkey Integration - CrewAI",
1106 "chunk_index": 59,
1107 "text": "llm\n=\nportkey_llm\n\n\n)\n\n\nSimple caching performs exact matches on input prompts, caching identical requests to avoid redundant model executions.\nCopy\nAsk AI\nfrom\n crewai \nimport\n Agent, \nLLM\n\n\nfrom\n portkey_ai \nimport\n createHeaders, \nPORTKEY_GATEWAY_URL\n\n\n\n\n# Configure LLM with semantic caching\n\n\nportkey_llm \n=\n LLM(\n\n\n model\n=\n\"gpt-4o\"\n,\n\n\n base_url\n=\nPORTKEY_GATEWAY_URL\n,\n\n\n api_key\n=\n\"dummy\"\n,\n\n\n extra_headers\n=\ncreateHeaders(\n\n\n api_key\n=\n\"YOUR_PORTKEY_API_KEY\"\n,\n\n\n virtual_key\n=\n\"YOUR_OPENAI_VIRTUAL_KEY\"\n,\n\n\n config\n=\n{\n\n\n \"cache\"\n: {\n\n\n \"mode\"\n: \n\"semantic\"\n\n\n }\n\n\n }\n\n\n )\n\n\n)\n\n\n\n\n# Create agent with semantically cached LLM\n\n\nresearcher \n=\n Agent(\n\n\n role\n=\n\"Senior Research Scientist\"\n,\n\n\n goal\n=\n\"Discover groundbreaking insights about the assigned topic\"\n,\n\n\n backstory\n=\n\"You are an expert researcher with deep domain knowledge.\"\n,\n\n\n verbose\n=\nTrue\n,\n\n\n llm\n=\nportkey_llm\n\n\n)",
1108 "metadata": {
1109 "source": "https://docs.crewai.com/en/observability/portkey",
1110 "title": "Portkey Integration - CrewAI"
1111 }
1112 },
1113 {
1114 "id": "1c39e8de-ae7b-4a2e-bb9a-3c189127d1f7",
1115 "source": "https://docs.crewai.com/en/observability/portkey",
1116 "title": "Portkey Integration - CrewAI",
1117 "chunk_index": 60,
1118 "text": "backstory\n=\n\"You are an expert researcher with deep domain knowledge.\"\n,\n\n\n verbose\n=\nTrue\n,\n\n\n llm\n=\nportkey_llm\n\n\n)\n\n\nSemantic caching considers the contextual similarity between input requests, caching responses for semantically similar inputs.\n\n\n\n7. Model Interoperability\n\n\nCrewAI supports multiple LLM providers, and Portkey extends this capability by providing access to over 200 LLMs through a unified interface. You can easily switch between different models without changing your core agent logic:\n\n\nCopy\nAsk AI\nfrom\n crewai \nimport\n Agent, \nLLM\n\n\nfrom\n portkey_ai \nimport\n createHeaders, \nPORTKEY_GATEWAY_URL\n\n\n\n\n# Set up LLMs with different providers\n\n\nopenai_llm \n=\n LLM(\n\n\n model\n=\n\"gpt-4o\"\n,\n\n\n base_url\n=\nPORTKEY_GATEWAY_URL\n,\n\n\n api_key\n=\n\"dummy\"\n,\n\n\n extra_headers\n=\ncreateHeaders(\n\n\n api_key\n=\n\"YOUR_PORTKEY_API_KEY\"\n,\n\n\n virtual_key\n=\n\"YOUR_OPENAI_VIRTUAL_KEY\"\n\n\n )\n\n\n)\n\n\n\n\nanthropic_llm \n=\n LLM(",
1119 "metadata": {
1120 "source": "https://docs.crewai.com/en/observability/portkey",
1121 "title": "Portkey Integration - CrewAI"
1122 }
1123 },
1124 {
1125 "id": "7206f8dd-afd2-4d71-aa15-a2a1af78a3ec",
1126 "source": "https://docs.crewai.com/en/observability/portkey",
1127 "title": "Portkey Integration - CrewAI",
1128 "chunk_index": 61,
1129 "text": "api_key\n=\n\"dummy\"\n,\n\n\n extra_headers\n=\ncreateHeaders(\n\n\n api_key\n=\n\"YOUR_PORTKEY_API_KEY\"\n,\n\n\n virtual_key\n=\n\"YOUR_OPENAI_VIRTUAL_KEY\"\n\n\n )\n\n\n)\n\n\n\n\nanthropic_llm \n=\n LLM(\n\n\n model\n=\n\"claude-3-5-sonnet-latest\"\n,\n\n\n max_tokens\n=\n1000\n,\n\n\n base_url\n=\nPORTKEY_GATEWAY_URL\n,\n\n\n api_key\n=\n\"dummy\"\n,\n\n\n extra_headers\n=\ncreateHeaders(\n\n\n api_key\n=\n\"YOUR_PORTKEY_API_KEY\"\n,\n\n\n virtual_key\n=\n\"YOUR_ANTHROPIC_VIRTUAL_KEY\"\n\n\n )\n\n\n)\n\n\n\n\n# Choose which LLM to use for each agent based on your needs\n\n\nresearcher \n=\n Agent(\n\n\n role\n=\n\"Senior Research Scientist\"\n,\n\n\n goal\n=\n\"Discover groundbreaking insights about the assigned topic\"\n,\n\n\n backstory\n=\n\"You are an expert researcher with deep domain knowledge.\"\n,\n\n\n verbose\n=\nTrue\n,\n\n\n llm\n=\nopenai_llm \n# Use anthropic_llm for Anthropic\n\n\n)\n\n\n\n\nPortkey provides access to LLMs from providers including:\n\n\n\n\nOpenAI (GPT-4o, GPT-4 Turbo, etc.)",
1130 "metadata": {
1131 "source": "https://docs.crewai.com/en/observability/portkey",
1132 "title": "Portkey Integration - CrewAI"
1133 }
1134 },
1135 {
1136 "id": "a06e1fd1-01a7-413f-b9ea-4a2563094a68",
1137 "source": "https://docs.crewai.com/en/observability/portkey",
1138 "title": "Portkey Integration - CrewAI",
1139 "chunk_index": 62,
1140 "text": "verbose\n=\nTrue\n,\n\n\n llm\n=\nopenai_llm \n# Use anthropic_llm for Anthropic\n\n\n)\n\n\n\n\nPortkey provides access to LLMs from providers including:\n\n\n\n\nOpenAI (GPT-4o, GPT-4 Turbo, etc.)\n\n\nAnthropic (Claude 3.5 Sonnet, Claude 3 Opus, etc.)\n\n\nMistral AI (Mistral Large, Mistral Medium, etc.)\n\n\nGoogle Vertex AI (Gemini 1.5 Pro, etc.)\n\n\nCohere (Command, Command-R, etc.)\n\n\nAWS Bedrock (Claude, Titan, etc.)\n\n\nLocal/Private Models\n\n\n\n\nSupported Providers\nSee the full list of LLM providers supported by Portkey\n\n\n\nSet Up Enterprise Governance for CrewAI\n\n\nWhy Enterprise Governance?\n\nIf you are using CrewAI inside your organization, you need to consider several governance aspects:\n\n\n\n\nCost Management\n: Controlling and tracking AI spending across teams\n\n\nAccess Control\n: Managing which teams can use specific models\n\n\nUsage Analytics\n: Understanding how AI is being used across the organization\n\n\nSecurity & Compliance\n: Maintaining enterprise security standards",
1141 "metadata": {
1142 "source": "https://docs.crewai.com/en/observability/portkey",
1143 "title": "Portkey Integration - CrewAI"
1144 }
1145 },
1146 {
1147 "id": "a4957f34-fbb0-4348-bbdb-60954c55f9d4",
1148 "source": "https://docs.crewai.com/en/observability/portkey",
1149 "title": "Portkey Integration - CrewAI",
1150 "chunk_index": 63,
1151 "text": "Usage Analytics\n: Understanding how AI is being used across the organization\n\n\nSecurity & Compliance\n: Maintaining enterprise security standards\n\n\nReliability\n: Ensuring consistent service across all users\n\n\n\n\nPortkey adds a comprehensive governance layer to address these enterprise needs. Let’s implement these controls step by step.\n\n\n1\nCreate Virtual Key\nVirtual Keys are Portkey’s secure way to manage your LLM provider API keys. They provide essential controls like:\n\n\nBudget limits for API usage\n\n\nRate limiting capabilities\n\n\nSecure API key storage\n\n\nTo create a virtual key:\nGo to \nVirtual Keys\n in the Portkey App. Save and copy the virtual key ID\nSave your virtual key ID - you’ll need it for the next step.\n2\nCreate Default Config\nConfigs in Portkey define how your requests are routed, with features like advanced routing, fallbacks, and retries.\nTo create your config:\n\n\nGo to \nConfigs\n in Portkey dashboard\n\n\nCreate new config with:\n\n\nCopy\nAsk AI\n{",
1152 "metadata": {
1153 "source": "https://docs.crewai.com/en/observability/portkey",
1154 "title": "Portkey Integration - CrewAI"
1155 }
1156 },
1157 {
1158 "id": "61463837-4746-4fb6-b4aa-d0cc7dc00e40",
1159 "source": "https://docs.crewai.com/en/observability/portkey",
1160 "title": "Portkey Integration - CrewAI",
1161 "chunk_index": 64,
1162 "text": "Go to \nConfigs\n in Portkey dashboard\n\n\nCreate new config with:\n\n\nCopy\nAsk AI\n{\n\n\n \"virtual_key\"\n: \n\"YOUR_VIRTUAL_KEY_FROM_STEP1\"\n,\n\n\n \t\"override_params\"\n: {\n\n\n \"model\"\n: \n\"gpt-4o\"\n // Your preferred model name\n\n\n }\n\n\n}\n\n\n\n\n\n\nSave and note the Config name for the next step\n\n\n3\nConfigure Portkey API Key\nNow create a Portkey API key and attach the config you created in Step 2:\n\n\nGo to \nAPI Keys\n in Portkey and Create new API key\n\n\nSelect your config from \nStep 2\n\n\nGenerate and save your API key\n\n\n4\nConnect to CrewAI\nAfter setting up your Portkey API key with the attached config, connect it to your CrewAI agents:\nCopy\nAsk AI\nfrom\n crewai \nimport\n Agent, \nLLM\n\n\nfrom\n portkey_ai \nimport\n PORTKEY_GATEWAY_URL\n\n\n\n\n# Configure LLM with your API key\n\n\nportkey_llm \n=\n LLM(\n\n\n model\n=\n\"gpt-4o\"\n,\n\n\n base_url\n=\nPORTKEY_GATEWAY_URL\n,\n\n\n api_key\n=\n\"YOUR_PORTKEY_API_KEY\"\n\n\n)\n\n\n\n\n# Create agent with Portkey-enabled LLM\n\n\nresearcher \n=\n Agent(",
1163 "metadata": {
1164 "source": "https://docs.crewai.com/en/observability/portkey",
1165 "title": "Portkey Integration - CrewAI"
1166 }
1167 },
1168 {
1169 "id": "1f67068d-51f1-47a1-8411-2341f5e9b6bc",
1170 "source": "https://docs.crewai.com/en/observability/portkey",
1171 "title": "Portkey Integration - CrewAI",
1172 "chunk_index": 65,
1173 "text": "portkey_llm \n=\n LLM(\n\n\n model\n=\n\"gpt-4o\"\n,\n\n\n base_url\n=\nPORTKEY_GATEWAY_URL\n,\n\n\n api_key\n=\n\"YOUR_PORTKEY_API_KEY\"\n\n\n)\n\n\n\n\n# Create agent with Portkey-enabled LLM\n\n\nresearcher \n=\n Agent(\n\n\n role\n=\n\"Senior Research Scientist\"\n,\n\n\n goal\n=\n\"Discover groundbreaking insights about the assigned topic\"\n,\n\n\n backstory\n=\n\"You are an expert researcher with deep domain knowledge.\"\n,\n\n\n verbose\n=\nTrue\n,\n\n\n llm\n=\nportkey_llm\n\n\n)\n\n\n\n\nStep 1: Implement Budget Controls & Rate Limits\n\nStep 1: Implement Budget Controls & Rate Limits\nVirtual Keys enable granular control over LLM access at the team/department level. This helps you:\n\n\nSet up \nbudget limits\n\n\nPrevent unexpected usage spikes using Rate limits\n\n\nTrack departmental spending\n\n\n\nSetting Up Department-Specific Controls:\n\n\nNavigate to \nVirtual Keys\n in Portkey dashboard\n\n\nCreate new Virtual Key for each department with budget limits and rate limits\n\n\nConfigure department-specific limits",
1174 "metadata": {
1175 "source": "https://docs.crewai.com/en/observability/portkey",
1176 "title": "Portkey Integration - CrewAI"
1177 }
1178 },
1179 {
1180 "id": "4a44b9d0-23b4-423c-94ce-0735ba941350",
1181 "source": "https://docs.crewai.com/en/observability/portkey",
1182 "title": "Portkey Integration - CrewAI",
1183 "chunk_index": 66,
1184 "text": "Navigate to \nVirtual Keys\n in Portkey dashboard\n\n\nCreate new Virtual Key for each department with budget limits and rate limits\n\n\nConfigure department-specific limits\n\n\nStep 2: Define Model Access Rules\n\nStep 2: Define Model Access Rules\nAs your AI usage scales, controlling which teams can access specific models becomes crucial. Portkey Configs provide this control layer with features like:\n\nAccess Control Features:\n\n\nModel Restrictions\n: Limit access to specific models\n\n\nData Protection\n: Implement guardrails for sensitive data\n\n\nReliability Controls\n: Add fallbacks and retry logic\n\n\n\nExample Configuration:\nHere’s a basic configuration to route requests to OpenAI, specifically using GPT-4o:\nCopy\nAsk AI\n{\n\n\n\t\"strategy\"\n: {\n\n\n\t\t\"mode\"\n: \n\"single\"\n\n\n\t},\n\n\n\t\"targets\"\n: [\n\n\n\t\t{\n\n\n\t\t\t\"virtual_key\"\n: \n\"YOUR_OPENAI_VIRTUAL_KEY\"\n,\n\n\n\t\t\t\"override_params\"\n: {\n\n\n\t\t\t\t\"model\"\n: \n\"gpt-4o\"\n\n\n\t\t\t}\n\n\n\t\t}\n\n\n\t]\n\n\n}",
1185 "metadata": {
1186 "source": "https://docs.crewai.com/en/observability/portkey",
1187 "title": "Portkey Integration - CrewAI"
1188 }
1189 },
1190 {
1191 "id": "d8aba31d-d874-48f3-a610-b249529bf3a1",
1192 "source": "https://docs.crewai.com/en/observability/portkey",
1193 "title": "Portkey Integration - CrewAI",
1194 "chunk_index": 67,
1195 "text": "\"strategy\"\n: {\n\n\n\t\t\"mode\"\n: \n\"single\"\n\n\n\t},\n\n\n\t\"targets\"\n: [\n\n\n\t\t{\n\n\n\t\t\t\"virtual_key\"\n: \n\"YOUR_OPENAI_VIRTUAL_KEY\"\n,\n\n\n\t\t\t\"override_params\"\n: {\n\n\n\t\t\t\t\"model\"\n: \n\"gpt-4o\"\n\n\n\t\t\t}\n\n\n\t\t}\n\n\n\t]\n\n\n}\n\n\nCreate your config on the \nConfigs page\n in your Portkey dashboard.\nConfigs can be updated anytime to adjust controls without affecting running applications.\nStep 3: Implement Access Controls\n\nStep 3: Implement Access Controls\nCreate User-specific API keys that automatically:\n\n\nTrack usage per user/team with the help of virtual keys\n\n\nApply appropriate configs to route requests\n\n\nCollect relevant metadata to filter logs\n\n\nEnforce access permissions\n\n\nCreate API keys through:\n\n\nPortkey App\n\n\nAPI Key Management API\n\n\nExample using Python SDK:\nCopy\nAsk AI\nfrom\n portkey_ai \nimport\n Portkey\n\n\n\n\nportkey \n=\n Portkey(\napi_key\n=\n\"YOUR_ADMIN_API_KEY\"\n)\n\n\n\n\napi_key \n=\n portkey.api_keys.create(\n\n\n name\n=\n\"engineering-team\"\n,\n\n\n type\n=\n\"organisation\"\n,\n\n\n workspace_id\n=\n\"YOUR_WORKSPACE_ID\"\n,",
1196 "metadata": {
1197 "source": "https://docs.crewai.com/en/observability/portkey",
1198 "title": "Portkey Integration - CrewAI"
1199 }
1200 },