the-illusion-of-multi-agent-advantages/smfr-dataset

Synthetic Multi-Hop Financial Reasoning (SMFR) Dataset

Dataset Description

The Synthetic Multi-Hop Financial Reasoning (SMFR) dataset contains synthetic stock trading analysis problems designed to evaluate multi-step reasoning and computational capabilities of large language models. Each problem presents historical stock price data for multiple companies and asks questions about investor trading strategies and portfolio performance.

Dataset Structure

The dataset is split into multiple files based on problem complexity (number of investors):

Total samples: 604 (588 training + 16 validation)

Data Format

Each sample is a JSON object with the following structure:

{
  "problem": "Problem text with stock price data and question",
  "answer": {
    "investor_dates": {
      "Alice": ["January 5, 2026", "January 8, 2026"],
      "Bob": ["January 6, 2026", "January 10, 2026"]
    },
    "comparison": {
      "Alice": "January 8, 2026",
      "Bob": "January 10, 2026"
    },
    "answer": ["Alice"]
  },
  "cot": "Step-by-step reasoning chain",
  "metadata": {
    "task_type": "stock",
    "seed": 1000,
    "num_instances": 2,
    "aggregation_op": "earliest_date",
    "task_params": {
      "target_percentage": 10.0,
      "price_type": "Open",
      "question_type": "reverse_target_sell"
    },
    "breadth": 2,
    "depth": 2
  },
  "generation_params": {
    "question_type": "reverse_target_sell",
    "aggregation": "earliest",
    "price_type": "Open",
    "target_percentage": 10.0
  }
}

Field Descriptions

Top-level Fields

• —`problem` (string): The complete problem text including:
• —Historical stock price data for multiple companies
• —Transaction history for each investor
• —The question asking about portfolio performance

• —`answer` (object): Structured answer containing:
• —investor_dates: Dictionary mapping each investor to their list of valid dates
• —comparison: Dictionary mapping each investor to their optimal date
• —answer: List of investor name(s) who achieve the best result (may be empty list for "None")

• —`cot` (string): Chain-of-thought reasoning showing step-by-step calculations

• —`metadata` (object): Generation metadata including task parameters and configuration

• —`generation_params` (object): Parameters used to generate this specific problem

Answer Structure

The answer field contains three levels of information:

1. 1.`investor_dates`: All valid dates when each investor can achieve the target profit
2. 2.Key: Investor name (e.g., "Alice", "Bob")
3. 3.Value: List of date strings in format "Month Day, Year"
4. 4.Empty list means the investor cannot achieve the target

1. 1.`comparison`: The optimal date for each investor
2. 2.For "earliest" questions: the earliest date from their valid dates
3. 3.For "latest" questions: the latest date from their valid dates
4. 4.null if the investor has no valid dates

1. 1.`answer`: Final answer to the question
2. 2.List of investor name(s) who have the best optimal date
3. 3.Empty list [] means no investor can achieve the target (answer is "None")
4. 4.Multiple names indicate a tie

Metadata Fields

• —`task_type`: Always "stock" for this dataset
• —`seed`: Random seed used for generation (for reproducibility)
• —`num_instances`: Number of investors in the problem
• —`aggregation_op`: How to compare investors ("earliestdate" or "latestdate")
• —`task_params`: Problem parameters
• —target_percentage: Target profit percentage (e.g., 10.0 for 10%)
• —price_type: Stock price type used ("Open" or "Close")
• —question_type: Type of question ("reversetargetsell" or "reversetargetbuy")
• —num_distractors: Number of distractor companies in the haystack
• —`breadth`: Number of companies involved in transactions (2-6)
• —`depth`: Number of transactions per investor (2-6, always even)

Problem Types

The dataset contains two types of questions:

1. Reverse Target Sell Questions

Scenario: Each investor has completed several buy transactions and holds shares in one remaining stock. The question asks: who can sell these remaining shares earliest/latest to achieve at least X% profit?

Example:

Each investor has completed several transactions and holds shares in one 
remaining common stock. Based on when they could sell these remaining shares 
to achieve at least 10% overall portfolio profit, who has the earliest 
possible sell date to reach this target?

2. Reverse Target Buy Questions

Scenario: Each investor has completed several transactions and has already sold shares in one stock but hasn't bought them yet. The question asks: who would have had to buy these shares earliest/latest to achieve at least X% profit when they sold?

Example:

Each investor has completed several transactions and has already sold shares 
in one common stock but has not yet bought them. Based on when they would 
have had to buy these shares to achieve at least 10% overall portfolio profit 
when they sold, who has the earliest possible buy date to reach this target?

Dataset Statistics

Problem Complexity Distribution

• —Breadth (companies per problem): 2-6 companies
• —Depth (transactions per investor): 2-6 transactions (always even for proper buy/sell pairing)
• —Investors per problem: 2-6 investors
• —Distractor companies: 2-6 companies (not involved in transactions, only in haystack)

Parameter Coverage

The dataset provides balanced coverage across:

• —Question types: 2 types (reversetargetsell, reversetargetbuy)
• —Aggregations: 2 types (earliest, latest)
• —Price types: 2 types (Open, Close)
• —Target percentages: 13 values from 10% to 200% in 15% increments

Each combination appears multiple times across different complexity levels.

Answer Distribution

• —Valid answers: Problems where at least one investor can achieve the target
• —"None" answers: Problems where no investor can achieve the target (limited to maintain dataset quality)
• —Ties: Problems where multiple investors achieve the same optimal date

Data Source

Stock price data is sourced from real historical prices using Yahoo Finance API (yfinance). The data covers a 30-day period with daily OHLCV (Open, High, Low, Close, Volume) information for the following companies:

• —Apple (AAPL)
• —Microsoft (MSFT)
• —Alphabet/Google (GOOG)
• —Nvidia (NVDA)
• —Adobe (ADBE)
• —Airbnb (ABNB)
• —Amazon (AMZN)
• —Baidu (BIDU)
• —Coca-Cola (COKE)
• —Dropbox (DBX)

Validation Set

The balanced_dataset_single_validate_fixed.jsonl file contains 16 held-out samples used for validation during dataset generation. These samples should be excluded when evaluating model performance on the n=2 and n=3 datasets.

Usage Examples

Loading the Dataset

import json

def load_dataset(filepath):
    samples = []
    with open(filepath, 'r') as f:
        for line in f:
            if line.strip():
                samples.append(json.loads(line))
    return samples

# Load a specific split
train_data = load_dataset('balanced_dataset_single_2_fixed.jsonl')
validation_data = load_dataset('balanced_dataset_single_validate_fixed.jsonl')

Extracting Problem and Answer

sample = train_data[0]

# Get the problem text
problem_text = sample['problem']

# Get the final answer
final_answer = sample['answer']['answer']  # List of investor names or []

# Get intermediate reasoning
investor_dates = sample['answer']['investor_dates']
comparison = sample['answer']['comparison']

# Get chain-of-thought
reasoning = sample['cot']

Understanding the Answer Format

The answer format is designed to prevent guessing and requires multi-step reasoning:

1. 1.Structured intermediate steps: Models must compute valid dates for each investor
2. 2.Comparison step: Models must identify the optimal date for each investor
3. 3.Final answer: Models must determine who has the best optimal date

A correct answer requires:

• —Correctly parsing stock price data
• —Accurately calculating portfolio profit for each possible date
• —Identifying all valid dates that meet the target
• —Comparing optimal dates across investors
• —Handling edge cases (no valid dates, ties)