0xkamal7/dv-misleading-charts

ChartLieDetector

ChartLieDetector is an original, provenance-tracked instruction-tuning dataset for visualization practitioners, data journalists, analysts, and model builders who need to audit chart integrity without a multimodal pipeline. It pairs serialized chart specifications and captions with construction-grounded distortion labels, minimal path-level corrections, claim support judgments, and plain-language descriptions of the data. No upstream dataset contains these exact examples because the honest specification, injected distortion, caption claim, correction, and label are generated together, making every target traceable to a known transformation rather than a subjective model annotation.

What this data produced

Model: TODO(dv-model) | Adaptive Data quality: TODO(dv-adaptive-data-quality-score)

The repository records neither a resolved Data Visualization model nor a completed Adaptive Data quality result. They remain explicit TODOs rather than inferred values.

Schema

Each line in train-tagged.jsonl is a JSON object. prompt contains a serialized chart specification, and completion is stored as a JSON-encoded string.

{
  "prompt": "string: audit instruction followed by CHART SPECIFICATION and serialized JSON",
  "completion": "string containing JSON with distortions, claim_supported, corrected_spec, and what_the_data_actually_shows",
  "shard": "dv_misleading_charts",
  "license": "cc0-1.0",
  "source": "Original (this work)"
}

After decoding completion, its exact shape is:

{
  "distortions": [
    {
      "mechanism": "string",
      "explanation": "string"
    }
  ],
  "claim_supported": true,
  "corrected_spec": {
    "changes": [
      {
        "op": "set",
        "path": "string",
        "value": "JSON value present for set operations"
      }
    ]
  },
  "what_the_data_actually_shows": "string"
}

claim_supported may be true or false. distortions and corrected_spec.changes may be empty. Correction values retain their native JSON type.

Real training example

This is the first row of data/dv-charts/train-tagged.jsonl. The completion is decoded only for readability; its values are copied from the file.

{
  "prompt": "You are a data-visualization integrity auditor. Audit the serialized chart specification and its caption/claim. Return strict JSON with exactly four fields: distortions (each with mechanism and explanation), claim_supported (boolean), corrected_spec (the minimal path-level changes needed to make the chart and caption honest), and what_the_data_actually_shows. Do not infer a distortion that is not present.\n\nCHART SPECIFICATION:\n{\"annotations\": [], \"bar_width_encoding\": \"proportional_to_bin_width\", \"caption\": \"Most observations are in the 3 to 4 bin.\", \"chart_subtype\": \"histogram\", \"chart_type\": \"bar\", \"series\": [{\"data\": [{\"count\": 58, \"end\": 1, \"start\": 0}, {\"count\": 52, \"end\": 3, \"start\": 1}, {\"count\": 20, \"end\": 4, \"start\": 3}, {\"count\": 69, \"end\": 7, \"start\": 4}, {\"count\": 40, \"end\": 8, \"start\": 7}], \"name\": \"Observations\"}], \"title\": \"Observed value distribution\", \"x_axis\": {\"field\": \"interval\", \"scale\": \"linear\"}, \"y_axis\": {\"field\": \"count\", \"range\": [0, 70], \"scale\": \"linear\", \"tick_spacing\": \"proportional_to_value\", \"ticks\": [0.0, 17.5, 35.0, 52.5, 70.0]}}",
  "completion": {
    "distortions": [],
    "claim_supported": false,
    "corrected_spec": {
      "changes": [
        {
          "op": "set",
          "path": "caption",
          "value": "The 4 to 7 bin has the largest count (69); bin widths vary and must be shown proportionally."
        }
      ]
    },
    "what_the_data_actually_shows": "The 4 to 7 bin has the largest count (69); bin widths vary and must be shown proportionally."
  }
}

This example is deliberately useful: the chart specification is not distorted, but its caption is unsupported. The label therefore leaves distortions empty while correcting the claim.

Composition and per-shard licensing

The composition below describes the publishable tagged training file recorded by artifacts/dv-charts-provenance.json.

Published training rows: 3,204. Compilation license: CC0-1.0.

Every published row retains shard, source, and license. The separate holdout is evaluation-only and is not part of the training composition.

Curation methodology

1. 1.Honest chart first. A fixed random seed creates bar, line, area, scatter, and pie specifications with underlying values, axes, annotations, context, and a caption. Ground-truth summaries are calculated directly from those values, including correlations where relevant.
2. 2.Injected-distortion catalog. The builder applies a known transformation selected from truncated y-axis, inconsistent bin widths, dual axes with mismatched scales, area encoding for a linear quantity, cherry-picked date range, missing baseline, omitted categories, and nonuniform tick spacing. Each injection returns both an explanation and the exact path-level operation needed to restore the honest specification.
3. 3.Negative controls. The raw construction reserves 30% of examples as honest charts so a model is trained not to hallucinate a distortion. Claim support is varied independently, so an honest chart can still carry a false caption and a distorted chart can still make a data-supported claim.
4. 4.Deterministic validation. The builder asserts the output keys, unique distortion names per row, catalog membership, chart-type coverage, and the intended honest-example count before writing JSONL.
5. 5.Deduplication and split isolation. pipeline/dedup_split.py removes exact and near duplicate prompts with normalized hashing and 64-bit token-trigram SimHash before re-splitting. pipeline/provenance.py rejects exact or near train/holdout overlap at maximum Hamming distance 3. The provenance report checked 357 holdout rows and recorded 0 train/holdout leaks.
6. 6.Adaptive Data. Adaptive Data by Adaption Labs is the intended adaptation layer. The repository does not yet contain a verified Data Visualization quality score, so this card does not claim one.

Licensing and attribution

The chart specifications, injected transformations, labels, and packaging are original work released under CC0-1.0. There is no upstream source dataset to attribute. Every row nevertheless retains source: Original (this work) and license: cc0-1.0, and pipeline/provenance.py computes the compilation license as the most restrictive component. With one CC0 component, the compilation is CC0-1.0.

Adaptive Data by Adaption Labs is credited as the adaptation system.

Intended use

Use ChartLieDetector for supervised adaptation, chart-integrity evaluation, data-literacy tools, newsroom or analyst review aids, and controlled raw-versus-adapted ablations. Its text-only serialized format is useful when a model must reason about the semantics of a chart specification without image input.

Do not use it as the only gate for publication, regulatory disclosure, or consequential statistical claims. A human reviewer should inspect the underlying data, transformation pipeline, and rendered chart.

Limitations

• —The dataset reasons over serialized specifications, not pixels. It cannot teach visual detection of clipping, occlusion, typography, perceptual color problems, or renderer-specific layout failures.
• —Its fixed catalog does not cover every misleading-chart technique.
• —Programmatic distributions and captions are cleaner and more regular than real newsroom, scientific, or business charts.
• —A supported caption is not proof that the underlying data collection or causal interpretation is sound.
• —Exact and SimHash checks do not prove the absence of every semantic near duplicate.

Honest weakness: the text-only design makes distortion labels exact and reproducible, but it gives up the central perceptual difficulty of detecting a lie in a rendered image.

Reproducibility

Run pipeline/build_dv_dataset.py, then pipeline/dedup_split.py data/dv-charts, then pipeline/provenance.py configs/provenance-dv-charts.yaml. The fixed seed, chart generators, distortion catalog, validation assertions, provenance tags, license calculation, and leak report are included in this repository.

Created for the Adaption AutoScientist Challenge using Adaptive Data by Adaption Labs.