Tabular Data Understanding with LLMs: A Survey of Recent Advances and Challenges

TL;DR

Tables are everywhere in real business data — databases, spreadsheets, financial reports, scientific PDFs — but LLMs read text left-to-right while tables are inherently two-dimensional. To feed a table to a model you have to flatten it into a string (or render it as an image), and how you flatten it materially changes the answer: the same task can swing up to ~20-50% in accuracy just from formatting choices. This survey does three things: (1) builds a taxonomy of table representations (serialization, schema, image, specialized encoders), (2) catalogs the major tasks and benchmarks (Table QA, Table-to-Text, Fact Verification, Leaderboard Construction), and (3) names the three gaps holding the field back — benchmarks test mostly retrieval/math instead of real reasoning, models collapse on hierarchical/long/multi-table inputs (sub-50% where humans hit ~83-89%), and models don’t generalize across input formats. For someone building data-analysis agents, this is the field map of why “just paste the table into the prompt” fails and where the durable engineering work actually is.

Problem & Motivation

Here’s the concrete pain. A client hands you a 10-K financial report with hierarchical tables (merged cells, multi-level headers, footnotes), or a folder of Excel files, or a scientific paper with ablation tables, and asks for “a daily report on key sales activities.” You can’t pass a 2D grid to an LLM — its input is a 1D token stream. So you serialize the table into some text format: Markdown, HTML, JSON, CSV, a CREATE TABLE schema, key-value pairs, or you screenshot it and send the image to a vision model.

Three things go wrong, and they’re not edge cases:

1. The format you pick changes the answer. Sui et al. (2024) showed that omitting partition markers or reordering input dropped accuracy up to 20%; removing few-shot examples cost up to 50%. Performance varies ~5% just based on how close your format is to what the model saw in pretraining. This is brittleness you cannot debug away — it’s baked into how the model learned.

2. Real tables break the models. Benchmarks with hierarchical tables (HiTab), long embedded text + multiple tables (MULTIHIERTT), or multi-table joins (MMQA) report model accuracy below 50%, while humans hit 83-89%. The strongest model on MMQA (o1-preview) barely cleared 50% exact-match vs. ~89% human.

3. The benchmarks themselves test the wrong thing. Most “table reasoning” benchmarks are secretly retrieval: they were constructed by writing a SQL query or a math expression first, then back-translating it into a natural-language question. So a Text-to-SQL pipeline solves them. They don’t test diagnostic reasoning (“why did this metric drop?”), forecasting, or inferring user intent from a vague request.

Prior surveys covered prompting/training methods and transformer architectures. This one is deliberately task- and benchmark-centric: instead of “here are the methods,” it asks “what are we actually testing, what does it miss, and where’s the opportunity.”

What’s New (Core Contribution)

This is a survey, so “new” means new synthesis and framing, not a new model. The genuine contributions:

• A clean taxonomy of table input representations. Before: scattered ad-hoc choices (someone uses Markdown, someone uses JSON, no one compares). Now: four organized families — serialization, data schema, image, table encoder — with the tradeoffs of each made explicit (Figure 4). This is the most reusable part of the paper for a builder.
• A benchmark map organized by direction of difficulty. Before: benchmark lists. Now: Tables 1-3 tag each benchmark by what it pushes on — Domain Knowledge, Answer Format, Input Complexity, Reasoning Difficulty — so you can see the field’s trajectory and pick a benchmark that stresses your failure mode.
• Three named, evidenced gaps as a research agenda. Before: vague “tables are hard.” Now: (1) limited scope beyond math/retrieval reasoning, (2) lack of robustness on input complexity, (3) limited generalization across representations — each backed by specific numbers. This is the part that tells you where the moat is.
• Surfacing under-explored tasks like leaderboard construction (auto-aggregating result tables from papers) and serialization-to-serialization (format translation as a training task), which are practical and under-served.

Be honest about the limits: there’s no new method, no experiments the authors ran, no released artifact. Its value is as a map and a critique, not a tool.

How It Works (Technically)

Since this is a survey, the “mechanism” is the conceptual pipeline every table-LLM system follows plus the taxonomy of choices at each stage. Let’s trace one table all the way through.

Take the tiny table from Figure 4:

| Name | Acc. |
| Ours | 60.1 |

Stage 1 — Representation (the lossy step). The 2D grid must become model input. Your options:

• Serialization → flatten to text. Same data, many encodings:

  • Markdown: | Name | Acc. | |---|---| | Ours | 60.1 |
  • JSON: [{"Name": "Ours", "Acc.": 60.1}]
  • HTML: <table><tr><th>Name</th>...</table>
  • Data matrix: [[0,"Name","Acc."],[1,"Ours","60.1"]]
  • Index mapping: (0,0,"Name") (0,1,"Acc.") (1,0,"Ours") (1,1,60.1)
  • Text template: The method name Ours has an accuracy of 60.1. Each preserves different structure. Index mapping makes cell coordinates explicit (good for “what’s in row 2, col 1?”). Markdown is compact but loses merged-cell hierarchy. LaTeX’s \multicolumn is the one common format that natively encodes hierarchical headers — most others silently drop that relationship (Figure 5). That single fact is a practical gold nugget: if your tables have multi-level headers, serialize through LaTeX or you lose the hierarchy.

• Data schema → don’t send the data, send the blueprint: CREATE TABLE Method (Name VARCHAR PRIMARY KEY, Acc. FLOAT) or pd.DataFrame({...}). This sidesteps context-length limits entirely (you describe a million-row table in 5 lines), but it only works on clean, well-structured tables, and you lose the actual cell values — the model now has to generate code (SQL/pandas) to touch the data. Critical detail: drop the primary/foreign keys from the schema and accuracy falls off a cliff (Zhang 2023a, Chen 2024) — the keys are what let the model plan joins. Best results came from including 3 example rows alongside the schema.

• Image → render the table as a picture, feed to an MLLM. Preserves layout perfectly (merged cells, alignment, visual grouping). Zheng et al. (2024) fine-tuned LLaVA on table-structure tasks (cell extraction, cell location) and beat OCR+serialization. The catch is resolution: a big table at fixed resolution blurs, and accuracy degrades. Promising untested idea: image plus serialized text — structure from the picture, exact values from the text.

• Table encoder → a specialized neural module with row-wise and column-wise attention baked in, so 2D structure is preserved inside the model rather than flattened. TAPAS (row-based), TaBERT, tree/graph embeddings were the small-model era; now TableGPT2 (Su et al. 2024) bolts a table encoder onto 7B/72B base models during pretraining + fine-tuning, creating a “table foundation model” that beats generalist models and rivals task-specific ones. This is the highest-ceiling, highest-cost path — you’re modifying the architecture, not the prompt.

Stage 2 — Augmentation / sampling. Real tables overflow the context window. Instead of truncating (which silently drops the answer row), use embedding-based sampling: embed each row/column, then keep the rows nearest the question or nearest cluster centroids (centroid + semantic sampling beat naive approaches, Sui 2023). Add small augmentations — table size, column-keyword explanations — and you get better accuracy within the token budget.

Stage 3 — Model + tools. The (possibly sampled, augmented) representation goes to the LLM/MLLM, often with tool use: generate SQL or pandas, execute it, read the result back. For schema-based inputs this is the whole game.

Stage 4 — Task-specific output. Cell spans, computed numbers, free text, a SQL query, a verification label, or a structured table — depending on the task (TQA, Table-to-Text, Fact Verification, Text-to-Table, Leaderboard Construction).

The deepest insight running through all four stages: there is no universal representation. Performance is data-dependent and format-dependent, and most benchmarks pick a format for convenience, not because it’s optimal — which quietly biases every reported result.

Architecture & data flow

flowchart LR
  T[2D Table / DB / Spreadsheet] --> R{Representation choice}
  R -->|flatten to text| S[Serialization: MD / JSON / HTML / index-map]
  R -->|blueprint only| D[Data Schema: CREATE TABLE / DataFrame + PK/FK]
  R -->|render as picture| I[Image to MLLM]
  R -->|structural module| E[Table Encoder: row/col attention]
  S --> A[Sampling + Augmentation<br/>embedding-based row/col selection]
  D --> A
  I --> A
  E --> A
  A --> M[LLM / MLLM + tools<br/>SQL / pandas exec]
  M --> O[Task output:<br/>cells / number / text / SQL / label / table]
  O --> TASK[TQA · Table2Text · Fact Verify · Leaderboard]

Schematic of the paper's central finding: the *same* table+question gives different accuracy depending on representation and on small input-design choices (marker order, few-shot count). Toggle the knobs to see accuracy swing — illustrating why format is an engineering decision, not a cosmetic one. Numbers are illustrative of the ranges the survey cites (~5% format, up to ~20% ordering, up to ~50% few-shot), not measured data.

The algorithm, simplified

The survey doesn’t ship one algorithm, so here’s the representation + sampling core that captures the practical heart of building a table-LLM pipeline — the part the survey says matters most.

# Build the LLM input for a table question. The two decisions that move accuracy
# most are (1) which serialization and (2) which rows survive the token budget.

def build_table_prompt(table, question, fmt="markdown", token_budget=3000):
    # 1) Sampling: real tables overflow context. Don't truncate top-N (drops the
    #    answer row). Keep rows most relevant to the question via embeddings.
    q_vec = embed(question)                          # embed(x) -> vector
    scored = [(cos(q_vec, embed(row_to_text(r))), r) # semantic relevance per row
              for r in table.rows]
    kept, used = [], 0
    for score, row in sorted(scored, reverse=True):  # most-relevant first
        cost = est_tokens(row)
        if used + cost > token_budget: break          # respect the window
        kept.append(row); used += cost

    # 2) Augmentation: tell the model what it can't see (size, dropped rows).
    aug = f"[table has {len(table.rows)} rows; showing {len(kept)} most relevant]"

    # 3) Serialization: THE format that preserves hierarchy if you have it.
    #    LaTeX \multicolumn keeps multi-level headers; markdown/json drop them.
    body = serialize(table.header, kept, fmt=fmt)     # markdown|json|latex|index_map

    # For schema-style inputs you would instead emit CREATE TABLE ... PRIMARY KEY
    # (never omit the keys) + 3 example rows, and ask the model to write SQL.
    return f"{aug}\n{body}\n\nQuestion: {question}"

The whole survey is an argument that each of these three lines is a research problem, not a settled default.

Built on Prior Work

Results & Evidence

This is a survey — the “results” are the aggregated numbers it marshals to support its three gaps. The strongest, most decision-relevant figures:

• Shallow benchmarks are saturating. >80% on WikiTableQuestions and >93% on TabFact (Hussain 2025); >80% on TabFact/FEVEROUS fact verification (multiple). Translation: the easy, retrieval-shaped tasks are basically solved. Don’t build a product whose moat is “we answer WikiTableQuestions-style queries.”
• Complex inputs crater. HiTab and MULTIHIERTT: models <50%, humans ~83% on MULTIHIERTT. MMQA multi-table: o1-preview ~50% exact-match, humans ~89%. Translation: hierarchical, long-context, and multi-table tables are an open problem — that’s where defensible work lives.
• Format sensitivity is large and real. ~5% swing from pretraining-alignment; up to 20% from input ordering/partition markers; up to 50% from few-shot removal (Sui 2024). Translation: your prompt-engineering of table format is worth more than picking a slightly better model.
• Schema needs its keys. Removing primary/foreign keys sharply degrades schema-based performance; 3 example rows is the sweet spot (Zhang 2023a, Chen 2024).
• Images can win when fine-tuned on structure tasks (Zheng 2024 beat OCR+serialization with LLaVA), but degrade with resolution on large tables.

What the evidence does NOT establish: Because the authors ran no experiments, every number is borrowed and uses different models, formats, and metrics, so they aren’t directly comparable — you can’t conclude “JSON beats Markdown by X%” from this paper. The “benchmarks are secretly retrieval” critique is well-argued but qualitative. And the field moves fast: a Jul-2025 snapshot will date quickly on absolute model numbers (the gaps, though, are structural and will outlast the specific scores).

How You’d Use It

For an AI services company, this paper is a risk map and a scoping tool for any “chat with your data / documents / spreadsheets” engagement.

• Scoping & expectation-setting. When a client says “answer questions over our financial reports,” use the three gaps to triage: Is this retrieval (easy, ship it) or reasoning/hierarchy/multi-table (hard, price accordingly)? You can now tell a client before the SOW that hierarchical multi-table QA is a sub-50% problem, not a weekend of prompt engineering.
• Representation as the first lever. In a data-QA agent, the highest-ROI knob is the serialization + sampling layer, not the model. Standardize on a representation strategy: schema+SQL for clean databases, LaTeX/HTML serialization for hierarchical tables, image+text for messy PDFs. This is concrete, cheap to implement, and the survey says it can move accuracy 20-50%.
• Multi-agent fit (your ARC MAS background). This maps naturally to a small agent pipeline: a Representation agent that picks/produces the right format, a Sampler agent that fits the table to budget, a SQL/pandas tool agent that executes, and a Verifier agent (Fact Verification framing) that checks the answer against the source cells. The survey’s task taxonomy basically hands you the role decomposition.
• A productizable offering: leaderboard / table extraction. The under-served “leaderboard construction” task (extract (Task, Dataset, Metric, Score) tuples from documents) is a real document-intelligence product — competitive intel, financial data extraction, research aggregation — with low competition.
• Honest sales positioning. Knowing that benchmarks oversell (“93% on TabFact!”) lets you cut through vendor hype and not over-promise on your own demos.

Build Your Own (Minimal Recipe)

The smallest system that captures ~80% of the value is a representation-aware Table QA agent:

1. Ingest + normalize. Detect the table type. Clean tabular DB → keep as schema. Hierarchical/PDF table → render to image and extract structure. Reach for: pandas, pdfplumber/camelot for PDF tables, an OCR/structure model for images.
2. Representation router. A simple rules-or-LLM classifier that picks serialization (Markdown/LaTeX/JSON) vs. schema vs. image. Default: schema+SQL for clean DBs; LaTeX serialization when multi-level headers are detected; image+text fallback for messy ones.
3. Embedding-based sampler. Embed rows/columns, keep those nearest the question within the token budget, prepend an augmentation note about what was dropped. Reach for: any embedding model + numpy/faiss.
4. Execute via tools. For schema path, have the LLM emit SQL/pandas, run it, feed results back (ReAct-style loop). For serialized path, answer directly but cite cells.
5. Verifier pass. Re-check the answer against the source cells (the Table Fact Verification task as a self-check) before returning.

The two genuinely hard parts: (a) the representation router — deciding which format wins for a given table is unsolved, so start with heuristics and log failures; (b) hierarchical/multi-table handling — there’s no clean library; expect to write custom header-flattening and join-planning logic, and accept sub-50% ceilings on the gnarliest inputs.

How to Improve It

Limitations are the opportunity list:

1. Serialization-to-serialization training (the survey’s own suggestion). Fine-tune a model to translate JSON↔LaTeX↔Markdown↔index-map. A model robust to format by construction would kill gap #3. Testable: measure accuracy variance across formats before/after such fine-tuning.
2. Image + text fused input. The survey flags that nobody has systematically evaluated “structure from the image, values from the serialized text.” Run that ablation; it’s a clean, publishable (and productizable) experiment.
3. A reasoning-first benchmark. Build a Table QA set where questions cannot be back-translated from SQL — diagnostic (“why the drop?”), forecasting, and intent-inference (Spider 2 / Text2Analysis style). Filter out anything a Text-to-SQL baseline solves. This directly attacks gap #1.
4. Adaptive representation routing as a learned policy. Frame “pick the format” as a contextual-bandit/RL problem: state = table features (size, hierarchy depth, type), action = representation, reward = downstream answer correctness. The survey treats representation choice as fixed; making it learned and per-table is a real research contribution.
5. Multilingual Table-to-Text. The survey notes there are no non-English table-to-text benchmarks. Low-hanging fruit for impact and for serving non-US clients.

Glossary

• Serialization — flattening a 2D table into a 1D text string (Markdown, JSON, HTML, CSV, etc.) so an LLM can read it.
• Data schema representation — sending only the table’s structural blueprint (CREATE TABLE ..., or a pandas DataFrame definition) instead of its contents, so the model writes code to query it.
• Table encoder — a specialized neural module with row- and column-wise attention that preserves 2D structure inside the model rather than flattening it to text.
• Table foundation model — a large base model (e.g., TableGPT2, 7B/72B) that integrates a table encoder during pretraining/fine-tuning, specialized for tabular tasks.
• MLLM (multimodal LLM) — an LLM that also takes images; used here to read tables rendered as pictures.
• Text-to-SQL — converting a natural-language question into an executable SQL query over a database.
• Table QA (TQA) — answering a natural-language question using a table’s contents; output is cells, a number, or a short text span.
• Table-to-Text — generating a natural-language description/summary of a table (or a highlighted region of it).
• Table Fact Verification (TFV) — given a claim + a table, label it Supported / Refuted / Not Enough Info (a.k.a. table NLI).
• Leaderboard construction — automatically extracting (Task, Dataset, Metric, Score) tuples from result tables across papers to build a comparison.
• Hierarchical / multi-level table — a table with merged cells or nested headers (rows/columns grouped under super-headers), common in finance and science.
• \multicolumn — a LaTeX command that spans a header across multiple columns; one of the few serialization features that natively encodes header hierarchy.
• Embedding-based sampling — selecting which rows/columns to keep (within a token budget) by embedding them and keeping those most similar to the question or to cluster centroids.
• Primary/foreign keys (PK/FK) — schema fields that uniquely identify rows (PK) and link tables (FK); essential for the model to plan joins.
• Few-shot examples (shots) — example input/output pairs placed in the prompt; removing them here cost up to ~50% accuracy.
• Exact-match (EM) — a strict metric where the model’s answer must equal the gold answer exactly to count as correct.