Every failure in this post is silent. That is what makes them worth knowing
about.
A document that failed to parse is indistinguishable from a document that
contains nothing relevant.
No error appears. Nothing turns red. The system answers your question using
whatever it did manage to read, with exactly the same confidence it would have
had if it had read everything. This is the most dangerous failure mode in a
document AI system, and it is the reason parsing — not the model — is the
highest-leverage layer in the stack.
Here is what actually breaks, in roughly the order it will bite you.
1. The scan with no text layer
A PDF is not a text format. It is a page-description format, and some PDFs
contain only pictures of words.
Anything that went through a scanner, a fax, or a photocopier and back into a
PDF may have no extractable text at all. Without OCR, a naive extractor returns
an empty string — and an empty document does not raise an error. It simply
contributes nothing to every answer thereafter.
The specific cruelty is that scanned documents skew old, and old documents skew
important. The original signed agreement is far more likely to be a scan than
last week's draft.
2. The table that becomes a run of numbers
Tables carry meaning in two dimensions, and naive extraction has one.
A liability table with columns for category, cap and exclusions becomes,
after flattening, a sequence of words and numbers with the headers detached from
the values. Ask what the cap is for category three and the system has a pile of
numbers and no idea which one you mean. It will often give you one anyway.
This is the failure most likely to produce a confidently wrong answer about a
number, which is the worst kind of wrong answer to produce.
3. Merged cells and formatting as meaning
In a spreadsheet, layout frequently is the data.
A merged cell spanning four rows means "this applies to all four" to a human and
means one value and three blanks to an extractor. Bold means "total". A yellow
fill means "provisional". An indent means "sub-item of the thing above".
None of that survives naive extraction, and worse, some of it survives
incorrectly — values land against the wrong rows and the result looks like
clean, plausible data.
4. The email thread with five generations of quoted history
The same paragraph appearing eight times is not eight pieces of evidence.
Reply chains carry their own history. Extract a long thread naively and you get
the original message repeated once per reply, with signature blocks, disclaimers
and "On Tuesday, X wrote:" interleaved.
Two things go wrong. Retrieval scores the repeated passage as unusually
important, because repetition looks like significance. And the chronology
inverts — the newest content sits at the top, so a naive reader of the extracted
text can conclude the opposite of what the thread actually decided.
5. Nested containers
A zip inside an email attachment inside a shared folder is three layers away
from being a document.
Real estates are full of archives containing archives, PDFs with embedded
attachments, and files that are actually containers for other files. Anything
that stops at the first layer indexes the wrapper and misses the contents — and
again, silently.
6. Drawings, diagrams and slides that are pictures
A slide containing a picture of a table is, to an extractor, an image with a
title.
Engineering drawings without a vector layer, architecture diagrams, org charts,
process flows exported as images, screenshots pasted into documents — these
carry substantial information for a human and close to none for naive text
extraction.
What actually helps
Layout-aware parsing: using document structure to decide where a chunk begins
and ends.
Rather than splitting on a character count, the parser uses headings, table
boundaries and section hierarchy — so a clause stays attached to its heading and
a table stays a table. Thirty-plus format parsers, because a spreadsheet, a
scanned contract and an email thread each fail differently and each needs
handling on its own terms.
Chunk boundaries are a retrieval decision made at parse time. Get them wrong
and you have capped the quality of every answer the system will ever give,
regardless of how good the model is. No amount of model capability recovers a
clause that was split away from the heading that gave it meaning.
And what still does not work
We are not going to pretend this is solved.
Layout-aware parsing degrades on handwriting, low-quality scans, drawings
without a vector layer, and tables whose structure is carried only by visual
alignment. Recovery on a given corpus should be measured, not assumed.
We publish no parsing accuracy percentage, because we do not have one that would
mean anything. Accuracy is a property of your documents. The honest version of
that number is one produced against your estate during a pilot, and any vendor
offering you a global figure is offering you a number about somebody else's
documents.
The test to run before you buy anything
Take your five worst documents and hand them over.
Not representative documents. The worst ones — the scanned contract from 2011,
the spreadsheet with merged cells that only one person understands, the drawing,
the nested archive, the forty-message email thread.
Then ask a question whose answer lives inside the structure you just stressed.
What is the liability cap for category three. A flattened table cannot answer
it, and you will find that out in ninety seconds rather than four months.
Do this before the demo with the clean sample documents, not after. Every
system looks capable on a clean corpus. Yours is not a clean corpus, and the
question that matters is what happens when it is not.