Apppublic

LG-Anonym/VerifiableRewardsForScalableLogicalReasoning

sourceHugging Faceupdated 1y agoView on Hugging Face

0likes

App README

Metric Card for Verifiable Logic Rewards

This metric is part of the SLR framework (LG-Anonym/SLR-Bench) and provides rewards for logical reasoning tasks. THe reward model is grounded in the ILP (Inductive Logic Programming) paradigm, testing whether a given hypothesis (logic rule) solves a logical reasoning task. TO check for entailment, the logic rule is executed against a set of background knowledge and examples, ensuring automatic evaluation that is verifiable, transparent, and reproducible.

How it Works

—Input: The symbolic judge takes as input a candidate hypothesis (logic rule) and an executable validation program containing background knowledge and examples.
—Execution: The candidate rule is executed against the validation program using a Prolog interpreter.
—Correctness Criteria: The rule is considered correct if it entails all positive examples and rejects all negative examples.
—Metrics: The symbolic judge computes a range of evaluation metrics (detailed below).

Note: A local Prolog interpreter is required to execute validation programs.

Inputs

—predictions (list of str): Each prediction should be a Prolog rule like "eastbound(T) :- Body."
—references (list of dict): Each reference should contain:
—validation_program (str): Prolog program with background knowledge and examples
—evaluation_config (dict, optional): Configuration with:
—positive_predicate (str): Predicate identifying positive examples (default: "eastbound")
—negative_predicate (str): Predicate identifying negative examples (default: "westbound")

Metrics & Output Values

—accuracy (float): Proportion of predictions that correctly classify all examples (0.0 to 1.0)
—partial_score (float): Average proportion of correctly classified examples (0.0 to 1.0)
—syntax_score (float): Proportion of rules with valid Prolog syntax (0.0 to 1.0)
—detailed_results (list of dict): Per-example results with:
—is_correct (bool): Whether the rule correctly classifies all examples
—partial_score (float): Proportion of correctly classified examples
—syntax_valid (bool): Whether the rule has valid syntax
—error (str, optional): Any error messages from Prolog evaluation
—exec_time (float, optional): Execution time for evaluation

How to Use with the datasets library

python

from evaluate import load
from datasets import load_dataset

# Load the symbolic judge for logical reasoning
symbolic_judge = load("LG-Anonym/VerifiableRewardsForScalableLogicalReasoning")

# load dataset LG-Anonym/SLR-Bench
ds = load_dataset('LG-Anonym/SLR-Bench', 'v1-All')
ds_test = ds['test'][:5]

# Prepare the predictions and references
rules = ds_test['ground-truth rule']
references = [{'validation_program': p,
                'evaluation_config': {
                    "positive_predicate": "eastbound",
                    "negative_predicate": "westbound"
                }
               } for p in ds_test['validation program']]
# Compute the evaluation
r2 = symbolic_judge.compute(predictions=rules, references=references)
r2

Outputs

python

{'accuracy': 1.0,
 'partial_score': 1.0,
 'syntax_score': 1.0,
 'detailed_results': [{'is_correct': True,'partial_score': 1.0,'syntax_valid': True,'error': None,'exec_time1': 0.014362812042236328},
                      {'is_correct': True,'partial_score': 1.0,'syntax_valid': True,'error': None,'exec_time1': 0.012364625930786133},
                      {'is_correct': True,'partial_score': 1.0,'syntax_valid': True,'error': None,'exec_time1': 0.012273550033569336},
                      {'is_correct': True,'partial_score': 1.0,'syntax_valid': True,'error': None,'exec_time1': 0.012486696243286133},
                      {'is_correct': True,'partial_score': 1.0,'syntax_valid': True,'error': None,'exec_time1': 0.012131929397583008}]}

Examples

Example 1: Evaluating a Single Rule

python

from evaluate import load

symbolic_judge = load("LG-Anonym/VerifiableRewardsForScalableLogicalReasoning")

validation_program = """
eastbound(train0).
has_car(train0, car0_1).
car_num(car0_1, 1).
car_color(car0_1, white).
car_len(car0_1, short).
has_wall(car0_1, full).

westbound(train1).
has_car(train1, car1_1).
car_num(car1_1, 1).
car_color(car1_1, yellow).
car_len(car1_1, short).
has_wall(car1_1, full).
"""

predicted_rule = "eastbound(Train):- has_car(Train, Car1), car_color(Car1, white)."

results = symbolic_judge.compute(
    predictions=[predicted_rule],
    references=[{"validation_program": validation_program,
                 "evaluation_config": {
                     "positive_predicate": "eastbound",
                     "negative_predicate": "westbound"
                 }}]
)

print(results)

Output Example 1

python

{'accuracy': 1.0,
 'partial_score': 1.0,
 'syntax_score': 1.0,
 'detailed_results': [
     {'is_correct': True,
      'partial_score': 1.0,
      'syntax_valid': True,
      'error': None,
      'exec_time1': 0.012056350708007812}]
 }

Example 2: Evaluating Multiple Rules

python

correct_rule = "eastbound(Train):- has_car(Train, Car1), car_color(Car1, white)."
incorrect_rule = "eastbound(Train):- has_car(Train, Car1), car_color(Car1, green)."

results = symbolic_judge.compute(
    predictions=[correct_rule, incorrect_rule],
    references=[
        {"validation_program": validation_program},
        {"validation_program": validation_program}
    ]
)

print(results)

Example 3: Custom Evaluation Configuration

python

validation_program = """
% Custom problem
parent(john, mary).
parent(john, bob).
parent(alice, bob).
parent(susan, alice).

% Examples
grandparent(susan, bob).
not_grandparent(john, alice).
"""

rule = "grandparent(X, Y) :- parent(X, Z), parent(Z, Y)."

results = symbolic_judge.compute(
    predictions=[rule],
    references=[{
        "validation_program": validation_program,
        "evaluation_config": {
            "positive_predicate": "grandparent",
            "negative_predicate": "not_grandparent"
        }
    }]
)