Healthcare manufacturing · Quality · Caso de estudio
Turning tables pasted in as pictures, and values written in by hand, back into table data with its rows and columns alive
Lo que consiguesTake QbD quality documents that mix hand-written values and captured image tables, handle even merged cells and irregular layouts, and get back searchable, aggregatable table data with its rows and columns alive.
The quality-control department of a healthcare manufacturer. Quality documents are piled up in shared folders and filing cabinets — papers where design-stage experimental values, validation results, and measurements by condition are organized inside tables. A staff member can tell at a glance which value is written by hand in which cell, and which table is a picture pasted from a screen capture.
The problem is the system. The moment you try to search or aggregate those same documents, the system sees the table not as "characters" but as a "picture." Hand-written numbers aren't picked up, and a table pasted in as an image is just a single photo with its rows and columns gone. That gap was blocking the very starting line of putting quality data to use.
So this company settled on an order with Consilience. Before promising clever analysis, first prove the work of making the documents properly "readable." This is the record of that first stage — a customer PoC (Proof of Concept, a pilot project to verify whether it actually works).
Print, handwriting, and screen captures mixed on a single page
This company's quality documents mix three things on a single page: printed field labels, values written in by hand on site, and images where a table built on a screen was captured and pasted in as is. A representative example is QbD (Quality by Design, an approach that plans quality from the design stage onward) documents. Design, experiment, and validation data all go into tables — and parts of those tables are filled in by hand while others are pasted in as pictures.
- Quality documents piled up in shared folders and filing cabinets. Design-stage experimental values, validation results, and measurements by condition are organized inside tables
- QbD documents. Design, experiment, and validation data all go into tables
- Printed field labels — these are the pre-printed form
- Hand-written values — data filled in on site. Everyone's handwriting differs
- Images where a table built on a screen was captured and pasted in — a clump of pixels whose row-and-column structure is wholly gone
- The tricky cases: handwriting inside a single cell, several cells merged into one, and irregular layouts that don't follow a fixed grid
Nothing starts until someone reads each page and copies the values over
For a person, there's no problem at all. They read the hand-written value, gauge which column it belongs to, and follow the rows and columns with their eyes even when the table is inside a picture. But the moment you try to handle this document as data, you hit a wall. Handwriting and image tables can't be searched or aggregated. Until you restore a table pasted in as a picture back into a table, you can't pull out the value of even a single cell. In the end, someone has to look at the documents one page at a time and copy the values over by hand. Searching, analysis, and knowledge-building on top of quality documents only begin once that preprocessing is done.
If it were only printed characters, plain OCR (Optical Character Recognition, the technology that turns text inside an image into text) would be enough. The problem is that the moment handwriting is mixed in, the recognition difficulty rises sharply. When a printed label and a hand-written value sit together in the same cell, you first have to tell which is the pre-printed form and which is the data a person wrote in. The table structure is no easier either. If a cell holds handwriting, or several cells are merged, or the layout is irregular, simple OCR splits the table in the wrong places or pushes a value into the wrong cell and collapses.
A demo that only handles easy documents is meaningless. It's only useful once it can withstand the tricky documents of the real field.
Putting the quality documents into the "Extract sources" wizard
The strategy for this PoC was clear. Rather than promising everything at once, first validate whether document preprocessing works properly. The order is that this has to succeed before you stack the stage that connects meaning on top. So the demo aimed at that one branch — preprocessing — alone. The goal was to show it as a screen flow a person can follow along with, and at the end actually produce a structured output.
The preprocessing screen flow
Open "Extract sources" with ⌘⇧I
Open the wizard with the "Extract sources" button in the left sidebar, or the shortcut
Cmd+Shift+I(Windows:Ctrl+Shift+I). Drag the quality documents anywhere onto the window, or add them with "Add manually" → "Add files" / "Add folders". Each row gets a kind label: Document, PDF, Image.Press "Import"
The button wakes up once a workspace is open and at least one source is listed, and the run starts the moment you press it. The first stage on the live checklist, "Reading your files", runs the markdown conversion.
Read the per-file status badges
Rows for scans and image-heavy documents show an "OCR…" badge — a vision call is in flight. ✓ means the note was written. An amber "Attached" means it couldn't be turned into markdown and is a placeholder note with only the original attached, so hover the badge to see exactly why.
Open the converted note and check the table
Once ✓ "Your documents are ready" appears, open the note. Look at whether the table that was a picture came in as a table with its rows and columns alive, and whether the hand-written values landed in their own cells. Your originals are never touched: they sit under
attachments/, linked from the converted note.
What we asked of the preprocessed tables
"Is this cell's value a printed label, or something a person wrote in?"
→ Read areas where printed text and handwriting are mixed together,
distinguishing the hand-written values from the printed labels.
"Can I pull out one cell's value from this table that was pasted in as a picture?"
→ Recognize tables pasted in as images and restore the row, column,
and cell structure. A table that was a clump of pixels comes back
as table data you can work with again.
"What happened to the merged cells and the off-grid layout?"
→ Handwriting inside a cell, merged cells, and irregular layouts are
handled separately. The table isn't split in the wrong places and
values aren't pushed into the wrong cells.Preprocessing was handled in these three branches: restoring the structure of image tables, distinguishing and recognizing areas where print and handwriting are mixed, and separately handling the tricky cases. The third was pulled out on its own because leaving it alone would land you with a "demo-only" result that works only on clean documents.
These are answers to "does it work," not to "how well does it work." What this stage confirmed is the sight of scattered hand-written and in-picture values getting organized into table data you can search and aggregate. What percentage of the recognition is correct was not pinned down here.
What changes, and what isn't there yet
| Before | What this PoC demonstrated |
|---|---|
| Hand-written numbers aren't picked up by search | Printed labels and hand-written values are distinguished and recognized |
| A table pasted in as an image is a single photo with its rows and columns gone | The row, column, and cell structure is restored, turning it back into table data you can pull a single cell's value out of |
| On merged cells and irregular layouts, simple OCR pushes values into the wrong cells and collapses | The tricky cases are handled separately, so it doesn't end as a demo that only works on clean documents |
| Someone has to look at the documents one page at a time and copy the values over by hand before search or analysis begins | A structured output comes out, organized into table data you can search and aggregate |
- Confirmed (fact): demonstrated the preprocessing screen flow for quality documents mixing handwriting and image tables, and generated a structured final output
- Unmeasured: quantitative performance such as recognition accuracy and processing speed — not yet measured
- Future plan (not started): the stage of connecting the structured data into an ontology and knowledge graph
One fact was confirmed in this customer PoC. Against quality documents mixing handwriting and image tables, we demonstrated the preprocessing screen flow and produced a structured final output on an actual screen.