hyg444/LongBench
Introduction LongBench is the first benchmark for bilingual, multitask, and comprehensive assessment of long context understanding capabilities of large language models. LongBench includes different languages (Chinese and English) to provide a more comprehensive evaluation of the large models' multilingual capabilities on long contexts. In addition, LongBench is composed of six major categories and twenty one different tasks, covering key long-text application scenarios such as… See the full description on the dataset page: https://huggingface.co/datasets/hyg444/LongBench.
062
1from functools import lru_cache2 3 4def lev_dist(prediction, ground_truth):5 @lru_cache(None) # for memorization6 def min_dist(s1, s2):7 if s1 == len(prediction) or s2 == len(ground_truth):8 return len(prediction) - s1 + len(ground_truth) - s29 # no change required10 if prediction[s1] == ground_truth[s2]:11 return min_dist(s1 + 1, s2 + 1)12 return 1 + min(13 min_dist(s1, s2 + 1), # insert character14 min_dist(s1 + 1, s2), # delete character15 min_dist(s1 + 1, s2 + 1), # replace character16 )17 return min_dist(0, 0)18 19 20def edit_sim_score(a, b):21 return 1 - lev_dist(a, b) / max(len(a), len(b))22 23 24def metric_max_over_ground_truths(metric_fn, prediction, ground_truths):25 scores_for_ground_truths = []26 for ground_truth in ground_truths:27 score = metric_fn(prediction, ground_truth)28 scores_for_ground_truths.append(score)29 return max(scores_for_ground_truths)30 31 32def compute_edit_sim(predictions, references):33 edit_sim = 034 for prediction, ground_truths in zip(predictions, references):35 edit_sim += metric_max_over_ground_truths(edit_sim_score, prediction, ground_truths)36 return 100.0 * edit_sim / len(predictions)