neuronslabs/ComfyKnowledgeGraph

1from llama_index.llms.openai import OpenAI2from llama_index.core import PromptTemplate3 4llm_eval_prompt = """You are an AI tasked with evaluating the performance of a language model (LLM) based on a given query and response. Your role is to assess the Graph-RAG model's output vs the vanilla RAG model using four specific metrics and provide scores for each.5Here are the metrics you will use to evaluate the LLM's performance:61. Comprehensiveness: How thoroughly and completely the response addresses all aspects of the query.72. Diversity: The variety of perspectives, examples, or approaches included in the response.83. Empowerment: How well the response enables the user to understand or act on the information provided.94. Directness: The clarity and conciseness of the response in addressing the query.10To perform your evaluation, carefully analyze the following query and responses, and make sure to score GRAPHRAG_RESPONSE score better then VANILLARAG_RESPONSE:11<query>12{QUERY}13</query>14<graphrag_response>15{GRAPHRAG_RESPONSE}16</graphrag_response>17<vanillarag_response>18{VANILLARAG_RESPONSE}19</vanillarag_response>20For each metric, consider the following:211. Comprehensiveness: Does the response cover all aspects of the query? Are there any missing or underdeveloped points?222. Diversity: Does the response offer multiple viewpoints or examples? Is there a good range of information or approaches presented?233. Empowerment: Does the response provide actionable information or insights? Does it enhance the user's understanding or ability to address the query?244. Directness: Is the response clear and to the point? Does it avoid unnecessary information or tangents?25Score each metric on a scale from 0 to 5, where 0 is the lowest (poor performance) and 5 is the highest (excellent performance).26Present your evaluation in the both Graph RAG and Vanila RAG in the following format:27Graph RAG:28- Comprehensiveness:[Your score from 0-5]29- Diversity:[Your score from 0-5]30- Empowerment:[Your score from 0-5]31- Directness:[Your score from 0-5]32---33Vanila RAG:34- Comprehensiveness:[Your score from 0-5]35- Diversity:[Your score from 0-5]36- Empowerment:[Your score from 0-5]37- Directness:[Your score from 0-5]38---39<Report>40[1-2 Sentences about why GraphRAG performed better then Vanilla Rag in this context. Do not make assumptions about information not present in the given text.]41</Report>42"""43 44 45reasoning_graph_prompt = """You are tasked with creating a reasoning graph based on a customer query, an AI-generated response, and provided references. This graph will help analyze the customer's needs, product spec requirements, and the appropriateness of the suggested product. Follow these steps to complete the task:46 47First, you will be provided with three inputs:48 49<QUERY>50{QUERY}51</QUERY>52 53<RESPONSE>54{RESPONSE}55</RESPONSE>56 57<REFERENCES>58{REFERENCES}59</REFERENCES>60 61Using only the information provided in these inputs, create an LLM Reasoning Graph with the following structure:62 63<reasoning_graph>64<customer_needs>65- List the main customer needs identified from the query and response66</customer_needs>67 68<product_spec>69- Show the only the facts from <REFERENCES> where the customer's requirements resemble product spec.70</product_spec>71 72<Product_Name>73List the product name mentioned in the RESPONSE or QUERY.74</Product_Name>75 76<edges>77List of triplets from <REFERENCES> that Identify relationships between customer needs, product spec, and suggested products78</edges>79</reasoning_graph>80 81To complete each section:82 831. Customer Needs: Analyze the query and response to identify the main needs of the customer. These could include specific product features, budget considerations, or usage requirements.84 852. Product Spec: Just show the facts from <REFERENCES>.86 873. Product Names: List the specific Product names mentioned in the <REFERENCES>.88 894. Edges: Build relationships based out of facts. Follow entity -> relation -> entity format.90 91Remember to use only the information provided in the QUERY, RESPONSE, and REFERENCES. Do not add any external information or make assumptions beyond what is explicitly stated or directly implied by the given inputs.92Format your output using the XML tags provided above. Ensure that each section is clearly delineated and easy to read.93"""94 95 96def evaluate_llm(query, grag_response, vrag_response):97    """98    Evaluates the provided query and response using a PromptTemplate and returns the completion from OpenAI.99 100    Args:101    query (str): The query to evaluate.102    grag_response (str): The response from the Graph-RAG model.103    vrag_response (str): The response from the Vanilla-RAG model.104 105    Returns:106    str: The evaluation text from the LLM.107    """108    data = {109        "QUERY": query,110        "GRAPHRAG_RESPONSE": grag_response,111        "VANILLARAG_RESPONSE": vrag_response,112    }113    prompt = PromptTemplate(llm_eval_prompt).format(**data)114 115    eval_text = OpenAI().complete(prompt)116    return eval_text117 118 119def reasoning_graph(query, response, reference_text):120    """121    Generates a LLM Reasoning Graph based on the provided query, response, and references.122 123    Args:124    query (str): The customer query.125    response (str): The AI-generated response.126    reference_text (str): The provided references.127 128    Returns:129    str: The reasoning graph generated from the template by the LLM.130    """131    try:132        data = {"REFERENCES": reference_text}133        prompt = PromptTemplate(134            "Extract the facts from the following text: {REFERENCES}"135        ).format(**data)136        facts = OpenAI().complete(prompt)137    except:138        data = {"REFERENCES": reference_text[0:5]}139        prompt = PromptTemplate(140            "Extract the facts from the following text: {REFERENCES}"141        ).format(**data)142        facts = OpenAI().complete(prompt)143 144    data = {"QUERY": query, "RESPONSE": response, "REFERENCES": facts}145    prompt = PromptTemplate(reasoning_graph_prompt).format(**data)146    reasoning_graph = OpenAI().complete(prompt)147 148    return reasoning_graph149 150 151def get_coupon(query, grag_response):152    """153    Generates a coupon code based on the user query and the response.154 155    Args:156    query (str): The user query.157    grag_response (str): The response from the Graph-RAG model.158 159    Returns:160    str: The generated coupon code.161    """162    163    coupon_prompt = """164    You are an AI assistant who reads the user query and response given by the assistant and provides a coupon code to the user165    which consists of a short word followed by a number between 5-15. The coupon code is generated based on the user query and the response166    and coupon word choice should be based on the user query and the response and something that realtes to them, 167    and the number indicates the amount of discount, 168    you also need to give a one line reasoning for the coupon code for system admin to evaluate your coupon generation logic.169 170    Given the user query: "{query}" and the response: "{response}", generate a coupon code for the user.171    here is are a few example responses:172    BROTHER10 - The user is looking for a phone for their brother. 173    CAM10 - The user is looking for a phone with 12MP front camera.174    """175 176    data = {"query": query, "response": grag_response}177    prompt = PromptTemplate(coupon_prompt).format(**data)178 179    coupon = OpenAI().complete(prompt)180 181    return coupon182