sxiong/TG_LLM_output

TG-LLM Output

This dataset contains inference outputs from the two-stage TG-LLM framework on the TGQA test set. TG-LLM performs temporal reasoning in two steps:

1. 1.Story-to-Temporal-Graph Translation (Story2TG): converts a story into a temporal graph.
2. 2.Temporal-Graph Reasoning (TGR): reasons over the predicted temporal graph to answer temporal questions.

The original TGQA dataset is available at `sxiong/TGQA`, and the source code is available in the `TG-LLM` repository.

Models

Both LoRA adapters use `meta-llama/Llama-2-13b-chat-hf` as the base model.

• —Story2TG: `sxiong/TG-LLM-TGQA-Story2TG-Llama-2-13B-LoRA`
• —TGR: `sxiong/TG-LLM-TGQA-TGR-Llama-2-13B-LoRA`

Dataset Configurations

The TGR prompts use the predicted temporal graphs produced in the first stage. The meta.TG field retains the reference temporal graph from TGQA for analysis; it is not the graph included in the TGR prompt.

Inference Configuration

• —Precision: BF16
• —Prompt format: plain text
• —In-context learning: disabled
• —Decoding: deterministic greedy decoding (do_sample=False, num_beams=1)
• —Story2TG maximum new tokens: 1,024
• —TGR maximum new tokens: 512

No sample reached its maximum generation length.

Data Fields

TGQA_story_TG_trans

• —id: Story identifier.
• —story: Input story.
• —TG: Reference temporal graph from TGQA.
• —prompt: Full prompt provided to the Story2TG model.
• —prediction: Temporal graph generated by the Story2TG model.

TGQA_TGR

• —id: Question identifier.
• —story: Source story.
• —question: Temporal reasoning question.
• —answer: List of reference answers.
• —prompt: Full TGR prompt containing the predicted temporal graph.
• —prediction: Generated reasoning trace and answer.
• —correct: Whether the parsed prediction exactly matches a reference answer.
• —meta: Additional reference data:
• —TG: Reference temporal graph from TGQA.
• —candidates: Candidate answers.
• —external knowledge: Temporal relations and arithmetic supplied in the prompt.
• —Q-Type: TGQA question-type index.

For correct, answers are compared case-insensitively after removing spaces. For question types 2 and 3, only the leading numeric value is compared, matching the TG-LLM evaluation protocol.

Results

All 3,316 TGR predictions were successfully parsed: Exact match (macro average) -- 77.5%.

Accuracy by Task

Loading the Data

After uploading this directory to a Hugging Face dataset repository, load either configuration with datasets:

from datasets import load_dataset

story_to_tg = load_dataset(
    "sxiong/TG_LLM_output",
    "TGQA_story_TG_trans",
    split="test",
)

tgr = load_dataset(
    "sxiong/TG_LLM_output",
    "TGQA_TGR",
    split="test",
)

Citation

@inproceedings{xiong2024large,
  title={Large language models can learn temporal reasoning},
  author={Xiong, Siheng and Payani, Ali and Kompella, Ramana and Fekri, Faramarz},
  booktitle={Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  pages={10452--10470},
  year={2024}
}