lorashen/Auto-SLURP
Auto-SLURP: A Benchmark Dataset for Evaluating Multi-Agent Frameworks in Smart Personal Assistant Repository for the paper Auto-SLURP: A Benchmark Dataset for Evaluating Multi-Agent Frameworks in Smart Personal Assistant requirements To test the multi-agent frameworks, you need to first install the framework according to the instruction of the framework. We have tested CamelAI, Langgraph, AgentLite, and AutoGEN. 1. start simulated servers cd server… See the full description on the dataset page: https://huggingface.co/datasets/lorashen/Auto-SLURP.
0108
1import re2wrong=03right=04wrongcount=05rightcount=06judgesent=re.compile(r"ChatCompletionMessage")7with open("eval.log") as f:8 lines=f.readlines()9 for line in lines:10 lowline=line.lower()11 if line.startswith("-----**"):12 if wrong>0:13 wrongcount+=114 elif right>0:15 rightcount+=116 else:17 print("error")18 wrong=019 right=020 elif judgesent.match(line):21 if lowline.find("wrong")!=-1 or lowline.find("incorrect")!=-1 or lowline.find("not completed")!=-1:22 wrong+=123 elif lowline.find("right")!=-1 or lowline.find("correct")!=-1:24 right+=125 elif lowline.find("content=''")!=-1:26 pass27 else:28 print("error")29 if wrong>0:30 wrongcount+=131 else:32 rightcount+=133acc=float(rightcount)/10034print(acc,"right",rightcount,"wrong",wrongcount)35