AxiomicLabs/LogicMark
LogicMark A procedurally generated benchmark for evaluating symbolic logic in language models. Each problem presents a set of variable equality/inequality premises and asks the model to identify which conclusion necessarily follows. Unlike knowledge-based benchmarks, LogicMark contains no facts a model could have memorised from pretraining. Every problem is generated fresh from abstract variable names (a, b, c, ...), so a model cannot pattern-match to training data - it must… See the full description on the dataset page: https://huggingface.co/datasets/AxiomicLabs/LogicMark.
LogicMark
A procedurally generated benchmark for evaluating symbolic logic in language models. Each problem presents a set of variable equality/inequality premises and asks the model to identify which conclusion necessarily follows.
Unlike knowledge-based benchmarks, LogicMark contains no facts a model could have memorised from pretraining. Every problem is generated fresh from abstract variable names (a, b, c, ...), so a model cannot pattern-match to training data - it must actually reason. This makes LogicMark a direct probe of intrinsic reasoning capability: the logical structure that has been built into the model's weights through training, independent of world knowledge or surface-level heuristics.
Evaluation is log-likelihood multiple-choice — no chain-of-thought, no prompting tricks. Models are scored purely on how well they assign probability to the correct completion.
Benchmark Results (10000 examples)
Evaluated using average log-likelihood over ending tokens, normalised by length. Random chance = 25%.
Task Format
If:
a = g
e != f
b != g
d = h
c = g
a != b
d = e
a = e
Then
A. c = h
B. c != h
C. c != e
D. b = hThe model must select the option that is logically entailed by the premises. Distractors include the flipped version of the correct answer and false statements drawn from the same variable set.
Hop Depth
Each problem is tagged with a hop depth — the minimum number of inference steps required to derive the correct answer from the premises:
- 1-hop: answer is directly stated as a premise
- 2-hop: requires one transitive step (e.g.
a=b,b=c→a=c) - 3-hop: requires two transitive steps
- 4-hop+: longer inference chains
The current dataset (1000 problems) targets the following distribution:
Graph Styles
Premises are generated using six equality graph topologies, each producing different reasoning patterns:
Generator
baseloggen_v2.py — full generator with hop depth control and equality type balancing.
from baseloggen_v2 import Config, generate_dataset, to_benchmark_format
cfg = Config(
limit=1000,
min_vars=4,
max_vars=8,
min_eq=2,
max_eq=5,
min_neq=1,
max_neq=3,
num_choices=4,
min_answer_hop=2,
target_hop_dist={1: 0.10, 2: 0.40, 3: 0.25, 4: 0.15, 5: 0.10},
styles=("chain", "star", "clusters", "tree", "bipartite", "mixed"),
seed=42,
)
dataset = generate_dataset(cfg)
formatted = [to_benchmark_format(p, i) for i, p in enumerate(dataset)]Key Config Parameters
Design Decisions
Equality type balance — correct answers are 50/50 sampled from = and != statements to prevent models from exploiting the observation that correct answers tend to be equality statements.
Exact hop targeting — when target_hop_dist is set, each bucket is generated with rejection sampling targeting exactly that hop depth, rather than relying on natural distribution (which heavily favours 2-hop).
Flipped distractor — the negation of the correct answer (e.g. a = b → a != b) is always included as a distractor to ensure the model can't win by ignoring inequality structure.
Dataset Format
{
"id": "symbolic_00042",
"domain": "Symbolic",
"context": "If:\na = b\nb != c\na = d\n\nThen",
"options": ["a != c", "d = c", "b = d", "a = c"],
"answer_index": 0,
"answer": "a != c",
"hop_depth": 2
}