frederickwang99/SoAyBench
SoAyBench by WangYC We've based SoAyBench creation on AMiner. To really understand how well LLMs can use SoAPI, we need to make AMiner's basic SoAPIs available for LLMs to use. We also need a test set made up of academic (question, solution, answer) triplets for checking how they're doing. The tricky part is, academic data keeps changing fast – stuff like info on scholars and their publications. So, keeping a test set with fixed answers is tough. To tackle this, what we've… See the full description on the dataset page: https://huggingface.co/datasets/frederickwang99/SoAyBench.
SoAyBench
by WangYC
We've based SoAyBench creation on AMiner. To really understand how well LLMs can use SoAPI, we need to make AMiner's basic SoAPIs available for LLMs to use. We also need a test set made up of academic (question, solution, answer) triplets for checking how they're doing. The tricky part is, academic data keeps changing fast – stuff like info on scholars and their publications. So, keeping a test set with fixed answers is tough.
To tackle this, what we've done is clone AMiner's SoAPIs as they were at a certain moment (Sep 15th 2023). This way, we've got a static version of the service. From there, we create a matching test set that doesn't change.
[toc]
Dataset Overview
You can find 44 jsonl files in SoAyBench.
Each of the jsonl file contains 18 lines.
Each line is a query-answer data like :
{
"Query": "Query in Chinese",
"Query_en": "Query in English",
"Answer": "Answer to the Query",
"Base_Question_zh": "Template query in Chinese",
"Base_Question_en": "Template query in English",
"Inputs": "Information which serves as the inputs of the APIs",
"Outputs": "The key of the answer at the API's response",
"Entity_Information": "Information that is filled into the template query"
}For example:
{
"Query": "Mutual Information领域的Jean Barbier的代表作的pdf链接是？",
"Query_en": "What is the PDF link of the representative work of Jean Barbier in Mutual Information field?",
"Answer": "//static.aminer.cn/misc/pdf/NIPS/2018/5b3d98cc17c44a510f801b5c.pdf", "Base_Question_zh": "XX领域的XXX的代表作的pdf链接是？",
"Base_Question_en": "What is the PDF link of the representative work of XXX in XX field?",
"Inputs": "name, interest",
"Outputs": "pdf_link",
"Entity_Information":
{
"name": "Jean Barbier",
"organization": "International Centre for Theoretical Physics",
"interest": "Mutual Information"
}
}How to use AMiner APIs that is included in SoAyBench?
In addition to providing a substantial amount of QA data, SoAyBench also includes a set of SoAPI services, encompassing a total of 7 APIs from AMiner. SoAy has filtered and wrapped the input and output of the original APIs into 7 functions.
You can find this details in https://github.com/RUCKBReasoning/SoAy/model.py
You can try all these APIs with SoAy/api_test.py
class aminer_soay:
def __init__(self):
self.addr = 'https://soay.aminer.cn/'
def searchPersonComp(self, **kwargs):
personList = []
addr = self.addr + 'searchPerson'
headers = {
'Content-Type' : 'application/json'
}
searchKeyWordList = []
if 'name' in kwargs:
searchKeyWordList.append({
"operate": "0",
"wordType": 4,
"keyword": kwargs['name'],
"advanced": True,
"needTranslate": True
})
if 'interest' in kwargs:
searchKeyWordList.append({
"operate": "0",
"wordType": 2,
"keyword": kwargs['interest'],
"advanced": True,
"needTranslate": True
})
if 'organization' in kwargs:
searchKeyWordList.append({
"operate": "0",
"wordType": 5,
"keyword": kwargs['organization'],
"advanced": True,
"needTranslate": True
})
json_content = json.dumps({
"sort": [{'asc': False, 'field' : 'n_citation'}],
"searchKeyWordList": searchKeyWordList,
"needDetails" : True
})
response = requests.post(
url=addr,
headers = headers,
data = json_content
)
result = response.json()
for each in result['data']['hitList']:
# print(each)
try:
personList.append(
{
'person_id' : each['id'],
'name' : each['name'],
'interests' : [each['interests'][i]['t'] for i in range(min(len(each['interests']), 10))],
# 'nation': each['nation'],
'num_citation' : each['ncitation'],
'num_pubs': each['npubs'],
'organization' : each['contact']['affiliation']
}
)
except:
continue
return personList
def searchPublication(self, publication_info):
addr = self.addr + 'searchPublication'
pubList = []
headers = {
'Content-Type' : 'application/json'
}
json_content = json.dumps({
"query" : publication_info,
'needDetails' : True,
'page' : 0,
'size' : 10,
"sort": [{'asc': False, 'field' : 'n_citation'}],
})
response = requests.post(
url=addr,
headers = headers,
data = json_content
)
result = response.json()
for each in result['data']['hitList']:
try:
pubList.append({
'pub_id' : each['id'],
'title' : each['title'],
'year' : each['year']
})
except:
continue
return pubList
def getPublication(self, pub_id):
addr = self.addr + 'getPublication'
addr = wrapUrlParameter(addr, id = pub_id)
# addr = addr + '?AppCode=' + self.appcode + '&id=' + id
response = requests.get(url = addr)
result = response.json()['data'][0]['pub']
info_dict = {}
try:
info_dict['abstract'] = result['abstract']
except:
info_dict['abstract'] = 'paper abstract'
author_list = []
for each in result['authors']:
try:
author_list.append({'person_id' : each['id'], 'name' : each['name']})
except:
continue
if author_list != []:
info_dict['author_list'] = author_list
try:
info_dict['num_citation'] = result['num_citation']
except:
pass
try:
info_dict['year'] = result['year']
except:
pass
try:
info_dict['pdf_link'] = result['pdf']
except:
pass
try:
info_dict['venue'] = result['venue']
except:
pass
return info_dict
def getPersonInterest(self, person_id):
addr = self.addr + 'getPersonInterest'
addr = wrapUrlParameter(addr, id = person_id)
# addr = addr + '?AppCode=' + self.appcode + '&id=' + id
response = requests.get(url = addr)
try:
result = response.json()['data'][0]['data']['data']['data']
except:
return []
interest_list = [result[i]['t'] for i in range(len(result))]
return interest_list
def getCoauthors(self, person_id):
addr = self.addr + 'getCoauthors'
addr = wrapUrlParameter(addr, id = person_id)
response = requests.get(url=addr)
result = response.json()['data'][0]['data']['crs']
coauthorsList = []
for each in result:
try:
coauthorsList.append({
'person_id' : each['id'],
'name' : each['name'],
'relation' : each['relation']
})
except:
continue
# coauthorsList = [{'person_id' : result[i]['id'], 'relation' : result[i]['relation']} for i in range(min(len(result), 10))]
return coauthorsList
def getPersonPubs(self, person_id):
addr = self.addr + 'getPersonPubs'
addr = wrapUrlParameter(addr, id = person_id, offset = 0, size = 10, order = 'citation')
response = requests.get(url=addr)
result = response.json()['data'][0]['data']['pubs']
pub_list = []
for each in result:
try:
pub_list.append({
# 'abstract' : result[i]['abstract'],
'pub_id' : each['id'],
'title' : each['title'],
'num_citation' : each['ncitation'],
'year' : each['year'],
'authors_name_list' : [each['authors'][j]['name']for j in range(len(each['authors']))]
})
except:
continue
return pub_list
def getPersonBasicInfo(self, person_id):
addr = self.addr + 'getPersonBasicInfo'
addr = wrapUrlParameter(addr, id = person_id)
response = requests.get(url=addr)
result = response.json()['data'][0]['data']
# print(response)
info_dic = {
'person_id' : person_id,
'name' : result['name'],
'gender' : result['gender'],
'organization' : result['aff'],
'position' : result['position'],
'bio' : result['bio'],
'education_experience' : result['edu'],
'email' : result['email']
# 'ncitation' : result['num_citation']
}
return info_dic
Original Service
We list all the original AMiner APIs below, which you can use to create new applications.
searchPerson
Information
Path： /soay.aminer.cn/searchPerson
Method： POST
Description：
Examples： 1.Basic Searching
{
"query": "jiawei han",
"needDetails": true,
"page": 0,
"size": 10
}2.Complecated Searching
{
"searchKeyWordList": [
{
"operate": "0",
"wordType": 5,
"keyword": "University of Illinois at Urbanan",
"advanced": true
},
{
"operate": "0",
"wordType": 4,
"keyword": "jiawei han",
"advanced": true
}
],
"filters": [
{
"boolOperator": "3",
"type": "term",
"field": "gender",
"value": "male"
}
],
"page": 0,
"size": 10,
"needDetails": true,
"aggregations": [
{
"field": "gender",
"type": "terms",
"size": 2
},
{
"field": "nation",
"type": "terms",
"size": 10
},
{
"field": "lang",
"type": "terms",
"size": 10
},
{
"field": "h_index",
"type": "range",
"rangeList": [
{
"from": 0,
"to": 10
},
{
"from": 11,
"to": 20
},
{
"from": 21,
"to": 30
},
{
"from": 31,
"to": 40
},
{
"from": 41,
"to": 50
},
{
"from": 51,
"to": 60
},
{
"from": 61,
"to": 99999
}
],
"size": 1
}
]
}Parameters
Headers
Body
Return
searchPublication
Information
Path： /soay.aminer.cn/searchPublication
Method： POST
Description：
Examples： 1.Basic Searching
{
"query": "data mining",
"needDetails": true,
"page": 0,
"size": 10
}2.Complecated Searching
{
"needDetails": true,
"page": 0,
"size": 20,
"aggregations": [
{
"field": "keywords.keyword",
"size": 20,
"type": "terms"
},
{
"field": "authors.orgid",
"size": 20,
"type": "terms"
},
{
"field": "year",
"size": 100,
"type": "terms"
}
],
"filters": [
{
"boolOperator": 3,
"type": "term",
"value": "data structure",
"field": "keywords.keyword"
}
],
"searchKeyWordList": [
{
"advanced": true,
"keyword": "jiawei han",
"operate": "0",
"wordType": 4
},
{
"advanced": true,
"keyword": "Mining frequent patterns without candidate generation",
"operate": "0",
"wordType": 1
}
]
}Parameters
Headers
Body
Return
getPublication
Information
Path： /soay.aminer.cn/getPublication
Method： GET
Description：
Parameters
Query
Return
getPersonBasicInfo
Information
Path： /soay.aminer.cn/getPersonBasicInfo
Method： GET
Description：
学者个人基础信息
Parameters
Query
Return
getPersonlnterest
Information
Path： /soay.aminer.cn/getPersonlnterest
Method： GET
Parameters
Query
Return
getCoauthors
Information
Path： /soay.aminer.cn/getCoauthors
Method： GET
Description：
学者网络关系
Parameters
Query
Return
getPersonPubs
Information
Path： /soay.aminer.cn/getPersonPubs
Method： GET
Description：
学者论文信息
Parameters
Query