Fine-Tuning Vision-Language Model for Automated Engineering Drawing Information Extraction

TL;DR

Manufacturers have to read GD&T (Geometric Dimensioning and Tolerancing) data off 2D engineering drawings — the little boxed symbols that say “this hole must be round to within 0.05mm relative to datum A.” Doing it by hand is slow and error-prone, and OCR/object-detection pipelines choke on the weird symbols, rotated text, and composite tolerances. The authors fine-tune Florence-2, a small open-source vision-language model (VLM), on 400 drawings annotated by domain experts, and compare it against GPT-4o and Claude-3.5-Sonnet running zero-shot (no fine-tuning) on the same data. The fine-tuned Florence-2 wins on every metric: +29.95% precision, +37.75% recall, +52.40% F1, and a 43.15% drop in hallucination versus the best closed-source model. The headline lesson for anyone building AI products: for a bounded, repetitive extraction task, fine-tuning a small specialist model beats prompting a frontier generalist — cheaper to run, more accurate, and yours to own.

Problem & Motivation

The concrete pain: Every machined part ships with an engineering drawing covered in Feature Control Frames — standardized boxed annotations encoding geometric characteristics (flatness, position, concentricity…), a tolerance value, and datum references. Inspection, quality control, and assembly all depend on reading these correctly. A single misread tolerance can pass a defective part or scrap a good one.

Today this is done manually (“ballooning” — engineers hand-mark each feature) or with semi-automated tools like Mitutoyo MeasurLink. Both are slow, don’t scale to high-volume production, and introduce data-entry errors that cause expensive rework.

Why prior automation falls short: People have thrown classic ML at this — YOLO for object detection, Tesseract for OCR. These break on the things that make GD&T hard:

• Composite tolerances (stacked control frames),
• Modifiers (the little circled M, L, P symbols that change tolerance meaning),
• Non-standard text orientations (annotations rotated to fit the drawing),
• Large labeled-data requirements that nobody has for this niche.

OCR sees characters but not meaning; object detectors see boxes but not the symbol-to-value relationships. You need something that reads pixels and reasons about structure jointly. That’s exactly what a vision-language model does — it ingests an image and a text query and emits structured text. The open question the paper answers: do you need a frontier model for this, or can a small tuned one do better?

What’s New (Core Contribution)

This is an applied/empirical paper. The novelty is less a new algorithm and more a clean, decisive demonstration plus the assets to reproduce it.

• Small-tuned beats big-zero-shot, decisively. Before: the instinct is “use GPT-4o, it’s smartest.” Now: a 230M-parameter model fine-tuned on 400 drawings beats GPT-4o and Claude-3.5 by double digits on every metric. The model is ~700x smaller than frontier models yet wins.
• A GD&T-specific dataset + annotation schema. Before: no standardized, machine-readable GD&T extraction benchmark. Now: 400 drawings normalized to PNG, expert-annotated into JSON capturing the three GD&T components (geometric characteristic, tolerance, datum), with 14 GD&T symbols encoded as Unicode so language models can actually emit them.
• A domain data-augmentation strategy via multi-query. Before: augmentation usually means rotating/cropping images. Now: they augment by pairing each image with 1, 2, or 4 different text queries — teaching the model to extract under varied prompting, which turns out to drive the recall gains.
• A hallucination metric tailored to extraction. Before: “accuracy” hides false fabrications. Now: they explicitly define hallucination = 1 − precision = the fraction of emitted entries that don’t exist in ground truth — directly measuring the model making things up, which is the scary failure mode in QC.

Be honest about the scope: the “method” is standard full-parameter fine-tuning of an existing model. The contribution is the result (small specialist wins), the dataset, and the augmentation trick — not a new training algorithm.

How It Works (Technically)

The whole system is a supervised fine-tuning loop wrapped around Florence-2, plus a zero-shot baseline harness for the big models. Let’s trace it end to end.

What Florence-2 actually is. Florence-2 is a sequence-to-sequence VLM: a vision encoder (DaViT) turns the image into a grid of visual tokens, those get projected into the same embedding space as text tokens, and a transformer encoder-decoder consumes [visual tokens] + [text prompt tokens] and autoregressively generates an output string. Crucially it’s a unified model — the same architecture does captioning, detection, OCR, grounding, depending on the prompt. That generality is why it fine-tunes well to a new task: you’re just teaching it a new prompt→output mapping, not bolting on new heads.

Step 1 — Build the data. Each of the 400 drawings is converted to PNG. A domain expert writes the ground-truth JSON: a list of {geometric_characteristic, tolerance, datum} records, with symbols written in Unicode (⌖ for position, ⏥ for flatness, etc.). A CSV ties each image index to its query text(s) and its ground-truth JSON. Drawings range from 0 to 14 GD&T entries — wildly variable density, which matters later.

Step 2 — Augment by multiplying queries. Three datasets are built from the same 400 images:

• Dataset 1: each image + 1 query
• Dataset 2: each image + 2 queries
• Dataset 3: each image + 4 queries

More query phrasings per image = more training examples and more prompt diversity, without needing more drawings. Each dataset is split 80/20 train/val.

Step 3 — Full-parameter fine-tuning. All 230M parameters are updated (not LoRA, not adapters — the whole model). Setup:

• Hardware: a single consumer NVIDIA RTX 4090. (This is the commercial punchline — you can train this on one gaming GPU.)
• 30 epochs, batch size 1, FP16 mixed precision.
• AdamW optimizer with cosine learning-rate decay starting at 1×10⁻⁶, no warmup.
• Loss: Florence-2’s default cross-entropy over output tokens.

Let me demystify those training knobs, since they’re the ML internals:

• Cross-entropy loss here just means: at each output position, the model predicts a probability distribution over the vocabulary; the loss is −log(probability it assigned to the correct next token). Minimizing it = “make the right JSON token more likely.” It’s the standard next-token training objective.
• AdamW is the optimizer that adjusts weights. The “W” = decoupled weight decay: it shrinks weights toward zero separately from the gradient step, which regularizes (prevents overfitting) more cleanly than plain Adam. Useful when you only have 400 images and overfitting is a real risk.
• Cosine decay from 1e-6 is a very gentle, smoothly-decreasing learning rate — you take tiny careful steps because you’re fine-tuning a pretrained model and don’t want to blow away what it already knows. Learning rate follows a cosine curve down to ~0 over training.
• Batch size 1 + FP16 is purely a memory accommodation: a 24GB 4090 can’t fit big batches of a VLM, so they process one image at a time in half precision.

Step 4 — The closed-source baseline. GPT-4o and Claude-3.5 get the same images zero-shot: a structured prompt asking them to find GD&T symbols/tolerances/datums and emit JSON. No fine-tuning — the authors note it’s computationally infeasible and impractical to fine-tune those giant closed models for a niche task (and you mostly can’t anyway).

Step 5 — Inference + a clever cleanup hack. Fine-tuned Florence-2 runs on its validation split. Raw model output isn’t always perfectly-formed JSON, so they post-process with GPT-4o-mini purely to fix formatting (not content) before scoring. Final outputs are saved as per-image JSON.

Step 6 — Scoring. Predicted key-value pairs are matched against ground truth. A True Positive requires an exact match of both the key AND the value (a tolerance of 0.05 vs 0.5 is wrong). Then:

The four metrics, in plain English:

Note hallucination is just the mirror image of precision — but framing it as a named metric makes the QC stakes legible: in manufacturing, a hallucinated tolerance is a defect waiting to happen.

Architecture & data flow

flowchart TD
  A[400 2D drawings<br/>PDF/JPEG] -->|normalize| B[PNG images]
  B --> C[Expert annotation<br/>JSON: char/tolerance/datum<br/>14 symbols as Unicode]
  C --> D[CSV: image idx + queries + GT]
  D -->|augment: 1/2/4 queries per image| E[Dataset 1/2/3<br/>80-20 train-val split]
  E --> F[Florence-2 base 0.23B<br/>full-parameter fine-tune<br/>30 epochs, RTX 4090]
  F --> G[Fine-tuned Florence-2]
  B --> H[Zero-shot prompt]
  H --> I[GPT-4o / Claude-3.5<br/>baseline, no tuning]
  G --> J[Raw predictions]
  I --> J
  J -->|GPT-4o-mini fixes JSON formatting| K[Clean JSON per image]
  K --> L[Score vs GT:<br/>Precision / Recall / F1 / Hallucination]

Schematic of how one drawing flows through Florence-2: image → vision encoder → visual tokens fused with the text query → transformer decoder emits GD&T JSON token by token. Click to step through.

The algorithm, simplified

The “method” is a vanilla supervised fine-tuning loop — the value is in the data prep and the head-to-head. Here’s the core loop with the GD&T specifics exposed:

# Fine-tune Florence-2 to map (drawing image, query) -> GD&T JSON.
# model(image, prompt) -> generated_string  (Florence-2 seq2seq VLM)

def build_dataset(drawings, queries_per_image):
    # AUGMENTATION = same image, several query phrasings (1, 2, or 4)
    examples = []
    for d in drawings:                       # d.png, d.gt_json (expert labels)
        for q in d.queries[:queries_per_image]:
            examples.append((d.png, q, d.gt_json))   # gt_json uses Unicode GD&T symbols
    return examples

def finetune(model, examples, epochs=30):
    opt = AdamW(model.parameters(), lr=1e-6, weight_decay=0.01)
    sched = CosineDecay(opt, total_steps=epochs * len(examples))
    for epoch in range(epochs):
        for img, query, target_json in examples:        # batch size = 1
            with autocast(fp16=True):                    # fit on a 24GB 4090
                logits = model(image=img, prompt=query)  # [seq_len, vocab]
                # cross-entropy: -log P(correct JSON token) at every position
                loss = cross_entropy(logits, tokenize(target_json))
            loss.backward(); opt.step(); sched.step(); opt.zero_grad()
    return model

def score(model, val_set):
    TP = FP = FN = 0
    for img, query, gt in val_set:
        pred = json_fix(model.generate(img, query))   # GPT-4o-mini repairs formatting only
        TP += count_exact_matches(pred, gt)           # key AND value must match exactly
        FP += count_only_in(pred, gt)                 # hallucinated / wrong entries
        FN += count_only_in(gt, pred)                 # missed entries
    precision = TP / (TP + FP)
    recall    = TP / (TP + FN)
    return precision, recall, 2*precision*recall/(precision+recall), 1 - precision

The two non-obvious design choices that actually move the needle: (1) augmenting by query-multiplication (Dataset 3, four queries/image, wins big), and (2) the exact-match scoring that punishes near-misses, which is what makes “Florence-2 wins” a strong claim rather than a soft one.

Built on Prior Work

The lineage is honest: nothing in the training stack is invented. The paper’s place in the field is “here is a clean, reproducible proof that the small-tuned-specialist strategy wins for industrial document extraction,” plus a dataset others can build on.

Results & Evidence

The full table (validation set, all percentages):

Headline: Exp-3 beats the best baseline on each metric by +29.95% precision, +37.75% recall, +52.40% F1, and cuts hallucination by 43.15%. Note the clear dose-response: more queries per image → monotonically better across all four metrics. That’s strong evidence the augmentation strategy is doing real work, not noise.

Telling secondary finding: within Exp-3, as the number of GD&T entries per drawing rises, recall and F1 fall and hallucination climbs (Fig. 5). Dense, complex drawings are still hard — the model degrades gracefully but degrades. This is the honest limitation: a 14-entry drawing is much harder than a 2-entry one.

What the evidence establishes: for this task, with this annotation scheme and exact-match scoring, a tuned 230M model clearly beats zero-shot frontier models, trainable on one GPU.

What it does NOT establish — read these before you generalize:

• Unfair baseline framing. The big models are zero-shot; the small one is fine-tuned. That’s the paper’s point, but it does not show Florence-2 is “better than GPT-4o” — it shows fine-tuning beats not-fine-tuning. A few-shot or carefully prompt-engineered GPT-4o, or a GPT-4o fine-tune (where available), is the missing comparison.
• Tiny, single-source dataset. 400 drawings, one annotation team, no reported inter-annotator agreement, no held-out test set from a different source/company. Generalization to other drawing styles is unproven.
• The GPT-4o-mini cleanup confound. A frontier model touches Florence-2’s outputs to fix JSON. They say formatting-only, but there’s no ablation showing how much the cleanup helps, and the baselines may not get equal treatment.
• Absolute numbers are modest. The winner still hits only 51% recall and 23% hallucination. This is a strong relative result but not yet production-grade for unsupervised QC — you’d want a human in the loop.
• No latency/throughput numbers, no per-symbol breakdown, no statistical significance testing.

How You’d Use It

This paper is a template for a repeatable service offering, not just a manufacturing curiosity. The pattern — “fine-tune a small open VLM to beat frontier zero-shot on a client’s narrow document-extraction task” — applies far beyond GD&T.

Concrete plays for an AI services company:

• Drawing/spec digitization-as-a-service. Manufacturers, fabricators, and contract shops drown in legacy 2D drawings. Offer GD&T (or BOM, weld-symbol, P&ID-tag) extraction. The moat is the labeled dataset + tuned model, which you own per-vertical.
• Pre-fill for downstream automation. The authors name the targets: process planning, tool selection, part classification, assembly validation, QC. Your extracted JSON becomes the input to a CAM/ERP integration — that’s where the client ROI compounds.
• Drop-in to an agentic pipeline. In a multi-agent system, this is a specialized “perception/extraction tool” the orchestrator calls: agent receives a drawing → calls extract_gdt(image) → gets structured JSON → routes to a validation agent that flags low-confidence or dense drawings for human review (directly informed by the Fig. 5 finding that dense drawings hallucinate more).
• Cost/control argument for clients. A 230M model runs on a $1.5k GPU or cheap cloud instance — no per-token frontier API bills, no data leaving the client’s network (huge for defense/aerospace IP). “We fine-tune a private model on your drawings and it beats GPT-4o” is a genuinely strong pitch.

The realistic build effort is the data, not the training. Training is an afternoon on one GPU. Getting 400+ expertly-annotated drawings with a clean schema is the hard, valuable part — and the thing you can charge for.

Build Your Own (Minimal Recipe)

Smallest version that captures ~80% of the value:

1. Pick the model. microsoft/Florence-2-base (or -large) from Hugging Face. Or any small open VLM with a processor + generate API (Qwen2-VL-2B, PaliGemma are good 2024+ alternatives).
2. Define a tight JSON schema for your target fields. Keep it flat: a list of records. Encode any special symbols as Unicode so the tokenizer can emit them.
3. Annotate 200-400 examples. This is the job. Get domain experts, write a labeling guide, store image→JSON pairs. Measure inter-annotator agreement on a subset so you trust your ground truth.
4. Augment by query-multiplication. Write 2-4 paraphrased prompts per image (“Extract all GD&T entries.”, “List geometric characteristic, tolerance, and datum for each control frame.”, …). This is the paper’s cheapest, highest-leverage trick.
5. Fine-tune. Use Hugging Face Trainer or the official Florence-2 fine-tuning notebook. Full-parameter if it fits (it does on a 4090 at batch 1, FP16); otherwise LoRA. AdamW, lr ~1e-6, cosine decay, ~30 epochs, early-stop on val loss.
6. Add a JSON-repair step. Constrained decoding (e.g. outlines, jsonformer) is cleaner than the paper’s GPT-4o-mini cleanup and keeps everything local.
7. Score with exact-match P/R/F1 plus the hallucination = 1−precision framing so stakeholders see the fabrication risk.

The two genuinely hard parts: (a) getting trustworthy expert annotations at scale, and (b) handling dense/complex inputs where recall collapses — budget for a confidence threshold + human-in-the-loop instead of pretending it’s fully autonomous.

Libraries to reach for: transformers, peft (if LoRA), accelerate, outlines/jsonformer for structured output, datasets.

How to Improve It

Limitations are the roadmap. Concrete, testable ideas:

1. Fix the unfair baseline — then claim a fair win. Re-run GPT-4o/Claude with few-shot exemplars and chain-of-thought, and fine-tune GPT-4o where the API allows. If Florence-2 still wins on cost-per-correct-extraction, that’s the bankable result.
2. Attack the dense-drawing collapse. Fig. 5 shows recall dies on 10-14 entry drawings. Try region-based decomposition: detect control-frame bounding boxes first (a cheap detector), crop each, extract per-frame, then merge. Turns one hard 14-entry problem into 14 easy 1-entry ones.
3. Confidence-aware output for HITL. Have the model emit token log-probs or a self-rated confidence per entry; route low-confidence drawings to a human. This makes a 51%-recall model usable in production by being honest about what it doesn’t know.
4. Constrained/grammar decoding instead of GPT-4o-mini cleanup. Enforce the JSON schema at decode time. Removes the frontier-model dependency, removes the confound, and likely lifts precision for free.
5. Bigger, multi-source data + active learning. 400 single-source drawings is the ceiling on generalization. Pool drawings from multiple companies/standards (ISO vs ASME Y14.5), and use active learning — let the model flag the drawings it’s least sure about for expert labeling, so each new annotation buys maximum accuracy.
6. Swap in a stronger small VLM. Florence-2 base is from 2023. A 2B-class 2024 VLM (Qwen2-VL, InternVL2) fine-tuned the same way could close the absolute-accuracy gap while staying small enough to self-host.

Glossary

• GD&T (Geometric Dimensioning and Tolerancing) — the standardized symbolic language on engineering drawings that specifies allowable variation in part geometry (per ASME Y14.5 / ISO).
• Feature Control Frame (FCF) — the boxed annotation that bundles a geometric characteristic, a tolerance value, and datum references.
• Datum — a reference feature (plane, axis, point) that tolerances are measured relative to.
• VLM (Vision-Language Model) — a model that jointly takes an image + text and outputs text; here, drawing + query → GD&T JSON.
• Florence-2 — Microsoft’s small (0.23B/0.77B) open-source unified VLM; one architecture for captioning, detection, OCR, grounding.
• Zero-shot — running a model on a task with no task-specific training, relying only on pretraining + the prompt.
• Full-parameter fine-tuning — updating all of a model’s weights during training (vs. LoRA/adapters, which update a small added subset).
• LoRA / adapters — parameter-efficient fine-tuning: freeze the base model, train a few small extra matrices. Cheaper but sometimes less accurate than full fine-tuning.
• Cross-entropy loss — the standard next-token training objective: penalize the model by the negative log-probability it assigned to the correct token.
• AdamW — an optimizer (Adam with decoupled weight decay) that adjusts weights using gradient history while regularizing toward zero.
• Cosine learning-rate decay — schedule that smoothly lowers the learning rate along a cosine curve over training, for stable convergence.
• FP16 / mixed precision — using 16-bit floats to halve memory and speed up training, so a large model fits on a consumer GPU.
• Epoch — one full pass over the training dataset.
• Precision / Recall / F1 — correctness of what was emitted / completeness of what was caught / their harmonic mean.
• Hallucination (here) — fraction of emitted entries with no basis in ground truth = 1 − precision.
• True/False Positive/Negative (TP/FP/FN) — correctly emitted / wrongly emitted / missed items, under exact key-and-value matching.
• Ballooning — manually numbering and tabulating each feature on a drawing for inspection.