Hey r/research, I’m a software engineer working on extracting clean structured data from ugly scientific PDFs, think multi-level headers, merged cells, nested row labels, footnotes mixed into labels, repeated subheaders, broken text layers, and scanned physical documents.
There are some decent tools in this space, specifically I've been building on top of PyMuPDF, Camelot, GMFT/Table Transformer, and img2table. Each has different strengths, so I’m combining them in a hybrid pipeline instead and trying to heuristically select the best results on a table by table basis, axis by axis basis.
It’s working pretty well so far, and on clean tables the existing tools do most of the work by themselves, where it breaks down is gross PDFs, the ones that when you try to copy and paste from it chrome freaks out and the text comes out all wrong, or you cant select it at all.
Currently from my test set I can recover the complete physical structure of a nasty 54-row table from a medium res scanned image. This specific test is one that all of the tools I'm building on fail in isolation, but using them as an ensemble my system is recovering it correctly. One of the more interesting cases has a symptom score with nested Pain, Burning, Paraesthesiae, and Numbness underneath it, and that hierarchy comes through correctly instead of being flattened into unrelated rows.
I’m looking for more hard cases so I can find where it still under performs, any cases with ugly merged headers, interrupted sections, image-only scans, corrupted symbols, enormous tables, or layouts that make no sense are all welcome.
Unfortunately, since I'm an engineer not a researcher, I don't have any kind of journal access, so anything you want to share would need to be open access. DOI, PMID, or direct PDF link would be ideal.
Hey, if you read to this point, thank you, and please help me break my code.