Make OCR Great Again

This English version was machine-translated from the Chinese original. Technical terms may require verification.

I don't study OCR. I used to talk about it, but I thought about it in my head, but I just wrote it from the picture."Boom!"It's pretty good to come out, like the extraction of words from the big phone albums.

The questions in this article can also be addressedRG: From project practice to system approach、MCP (Model Context Protocol)How the concept of a relatively close read together is developed in different contexts.

And then I started studying Agent, and I realized that OCR sometimes gets stuck in the front of the entire system. The real world is not so clean..png、.pdf、.docx And all the strange formats are inputs that the system has to process. Scientific research can directly use the cleaned data sets, not engineering, and we have to get these things in.

The document is usually troublesome more than words. Where the table is, how the formula is kept, what the title and the text are, whether the reading order is not broken, whether the diagram and the annotated note are to be placed together, whether the picture on the page is to be cut, described or left with a reference. These issues, which were more like later, are now becoming part of OCR itself. Now OCR is not the same as it used to be, it's a fixed line used to clean up raw data and facilitate the back RAG.

OCR, not recognition, document resolution.

If it's just a text, a lot of numbers PDF should never have gone OCR. The text is drawn directly from PyMuPDF, pdfPlumber, cheap, fast, and does not re-identify the original clean text and introduce errors.

OCR is becoming interesting again because we start looking at documents as a structured object. The page is not a string of characters, but a mixture of text, title, formulae, tables, pictures, footnotes, header feet and reading order. If the model is just a piece of text, it's going to take a lot of effort to guess where it comes from.

So now the more accurate name of this thing is document parsing. It's going to tear the page and spell back what a machine can do and can be read. Markdown is a good exit, but not the only exit, and more valuable is the structural information it retains.

So the question becomes: what should this structure look like?

Beyond the model: what OCR at the level of delivery should produce

If downstream is search, RG or Agent, OCR's delivery cannot be just a Markdown that seems right. Markdown is more like a preview layer, and what really should be delivered is a document object that can be tracked, split, enhanced and indexed.

The big factory has given the reference answer.Google Document AI Layout Parser You keep titles, tables, formulae, lists and hierarchical relationships, and then you generate contact-awarechunds with ancestral titles.Markdown of Azure Content Understanding Saves the chapter, table, formulae, pictures, page numbers and directories in a visible way. The common judgement behind them is that the pure OCR text flattens the reading order and context, and that the retrieval system needs to know which section, which page, which chart or which table a section is related to.

There are several design lines that can be learned from it: elements need to be stable, position information, content generated separately from the original, structure cannot be flattened, complex elements need searchable explanations, and each result can go back to the original page. It's the complete hosting service that's abandoned because I need to be able to run locally and to customize the logic of the process. In an era when Data is all you need, it is not a good idea to outsource the data-cleaning pipe.

Different elements require different delivery criteria

Object	Minimum deliverables	Retrieval-oriented enhancements
Text	Paragraphs, lists, reading order, language and page numbers, dealing with hyphens, hyphens and header footers	Keep chapter path, paragraph roles, key entities and context, without all text being crushed into a paragraph
Chapter Structure	Title level, paternity, directory anchor and chapter range	Write the ancestral title to chunk mettadata, let"Methodology""Result"There's still a whole spectrum of such duplicate titles.
Formula	LaTeX or MathML, line/line type, formula number, original map and coordinates	Add a neighbouring definition, variable interpretation and searchable natural language interpretation and mark whether the interpretation is generated by models
Pictures and Charts	Original cropping, original drawings, OCCR, page numbers, bbox, and references to text	Generate descriptive text describing objects, coordinates, legends and main visible relationships; while retaining visual access to embedding
Table	Cell grid, row title, merge cell, unit, title, footnote and original coordinates	Also provide structured data and retrieval summaries, and write key words into natural languages, without taking into account the conclusions under the original table

Formulas are particularly susceptible to error. Only one formula picture is left, and text search is almost impossible to find; only LaTeX is kept and may be missing"What's this formula for?"semantics. It would be more prudent to tie together formulae, numbers, ex-ante interpretations and variables. User Search"How to calculate long-term incentive discounts", you should be able to hit the section where the formula is, not ask for a query to appear. \gamma。

Pictures and forms should not be left alone. ![](images/xxx.png)I'm sorry. At least keep the original drawing notes, the text in the diagram, the chapter to which you belong and the quotations in the body. For pictures without a note, a description can be generated using VLM.Docling's picture of the situation That's what it is. But keep the source tag because"What is clearly written in the figure?"and"The model thinks the map is a sign of something."Not the same kind of evidence.

This leads to one of the most important items on the entire list that I believe is: faithful restitution of the original language and search-oriented enhanced text, which must be two fields. Photo descriptions, table summaries, formulae interpretations can expand recall, but they cannot be disguised as original facts. You can never tell whether a search result is a document or a model illusion.

The page number and coordinates are not all in the files.

page_idx and bbox For PDF is mandatory field. Word documents do not have a reliable set of final pages that are not relevant to the rendering environment. A visible page break can be found in the file, but the page numbers that you see in Word will still change with font, paper size and printer drive. Word and HTML, EPA, Markdown are closer to a continuous stream document.

So positioning information must be downgraded: page breaks to page numbers and bbox, flow formats to structure paths and characters. And the downstream search is written accordingly."There's bbox, frame, no structure.", the different positioning branches are maintained in format.

Don't cut the pieces, then try to find the structure.

The cut-off for retrieval should occur after structural recovery. A cut of every 500 tokens can easily separate the title from the body, separate the tablehead from the data line, or lose the picture description.

More rational order is:

1. Recover the page elements and read order first
2. Creates chapter tree, element and page-crossing relationships
3. chunk by chapter, paragraph and element boundary
4. Attach each chunk ancestor title, location information, document version and element type
5. Generate separate chunk for pictures, tables and formulae while retaining their relationship with the body father cunk

Character accuracy remains important at this time, but it is not the final acceptance standard. A system that delivers should also answer these questions:

• Can you hit the right paragraph when searching for the subject of the chapter and bring back the full title path?
• Can you retrieve the corresponding rows, column headings, units and footnotes at the same time when searching for a condition in the table?
• When searching for a phenomenon expressed in a chart, can you find a picture by drawing or by generating a description and return to the original page?
• When searching for formulae, can you return formulae, variable definitions and adjacent interpretations, rather than isolated LaTeX?
• Can each result be traced to the source document, location and distinguishing between the original language and the content of the generation?
• When a page or element is not deciphered, can the system clearly report failure rather than producing a seemingly complete Markdown?

The last one I felt was the easiest to ignore. The failure of the analysis is acceptable, and silence fails because you don't even know what's wrong.

How to select the right tool

A simple comparison.

With this list above, I have chosen three comparable options. They're all in OCR, but the scenes are different.

	Docling	MinerU	Unlimited OCR
It's a product.	IBM Research Zurich / LF AI & Data	Shanghai AI Lab / OpenDataLab	100 degrees
Positioning	Multiformer Parser + Unique	Document Parsing Frame, Three Core Modes	End to Long Document VLM
Input	PDF, Office, HTML, EPA, mail, audio, video, ODF, XBRL, etc. 20+	PDF, Pictures, DOCX, PPTX, XLSX	Pictures and PDFs only
Internal structure	Trees,body Root + JSON pointer	Legacy output is a reading-ordered list; 3.0 from a single structure with a separate page by page	Model straight out of marked text
Long Document	Page-based Processing Unit	Page/Sliding Window Organization	R-SWA, dozens of pages available for joint input
Permission	MIT	3.1.0 Commencement"Customise permission based on Apache 2.0"	MIT
Mature	High, Langchain / LlamaIndex / Haystack Official Integration	High, big community.	Weak, access to Vllm, and so on.

MinerU is more specialized in PDF and complex layouts, and the original format supported by Docling is more extensive, and provides a unified expression. Unlimmed OCR is trying to solve the problem of continuity of long documents.

Many document resolution models still use pages as basic inputs, with cross-page paragraphs, cross-page tables and serial numbers to be added from the outside; Unlimed OCR R-SWA allows the model to see visual input at all times when it is generated, but only the nearest window is left for the generated text, so KV Cache is pressed near a fixed limit and dozens of pages can be read out at once. It has been able to deploy through Transformers, vLLM and SSGLang, but the project itself has been unable to enter production directly.

DOCX, XLSX, PPTX are essentially ZIP containers, containing XML, media documents and the relationship between them. Most of the structure and content are already in the document, and the focus is on reading the original structure, rather than re-producing each page. Docling's fit in in that respect is more complete. MinernU has been more focused on PDF, pictures, formulae and complex layouts that require OCR or VLM involvement.

Excel has two uses, sometimes with very few tables, a watch schedule or board, and sometimes a data sheet. The former could be part of the knowledge base, while the latter should be placed in the database for use by the data centre department. For small-scale scheduling forms, I usually do the Docling forms. Chunker- Let a Sheet match a Table. I'll give you the big watch. HybridChunker Press token cut; it will customise the header when the table crosses the chunk.

First judge the document, then select the solver.

In this way, the logic of route is simple:

Input	Default Selection	When will you upgrade?
Office / HTML / EPUB / markdown	Docling format backend; straight out DoclingDocument Time to go. SimplePipeline	If the original structure is not read or format is not supported, turn PDF and go to the branch below
Digital Native PDF (text layer complete, layout general)	Docling standard Or MinerU. pipeline	Try VLM when there is a clear error in reading order, crossbar or complex table
Scan / Spectrum PDF	Open first OCR; available MinerU pipeline Or Docling. standard Sample	Try again when OCR is still unable to restore complex layouts vlm、Docling vlm
Re-formatted, complex tables, formulae intensive PDF	MinerU vlm, or Docling vlm	No upgrade. Watch the effect.

PDF has no available text layer is the first dividing line, but having a text layer does not mean that OCR is not necessarily used. I will extract the number of characters per page, printable character proportions and photo coverage, and then randomly render page contrasts; some PDF hidden text layers are not coded, wrongly positioned, and should be processed as a scanned copy. Nor are thresholds suitable for death, and the distribution of Chinese slides, double-bar papers and invoices varies considerably.

Docling: Processing XML and PDF should not be on the same pipe

Docling Official Hostage ServicePlease. Docling for IBM watsonxBottom of the line. docling-serve RET API. Given that there is no free access, I am generally deployed locally.

Docling selects the backend and Pipeline in the input format and adjusts the parameters inside Pipeline:

Layer	Options	What do you actually do?
Native Structure Format	Format backend + SimplePipeline	DOCX, PPTX, HTML, Markdown, etc. directly read paragraphs, titles, tables and photographic relationships in the file, output DoclingDocument；SimplePipeline Receive this result and execute uniform delivery without running PDF layout analysis, OCR and TableFormer
PDF / Photo Processing Pipeline	standard / vlm	standard Correspond StandardPdfPipeline, by linking specialized models such as layout, OCR, table structure, etc.;vlm Correspond VlmPipeline, convert visual language models to page-to-end versions suitable for scanning and unconventional layouts, but with higher speed, cost and generation error
standard Table Mode in	fast / accurate	fast Fits to simple forms or previews;accurate is the default mode for TableFormer, which is used when complex headers, cells are merged and error columns are larger

For direct generation DoclingDocument The form of the original structure.DocumentConverter Usually, it's automatic. SimplePipeline: it takes the result of the backend, and no longer runs the PDF set of page-level visual structures.SimplePipeline Keeps the uniform conversion result and interface to the enrichment, but whether it is worth a different description of the picture, which is to be opened on a task basis.

The rest of the parameters are mainly for PDF standard Pipeline:do_ocr The blogger says:force_ocr It is the enforcement of the entire page even if there is a text layer; normally the former is opened, and only when the text layer is hidden is damaged.do_table_structure Controls whether or not to restore the table grid; columns can be closed when they are merged by error do_cell_matching, the cell text predicted by the table model.

The code, formulae, photo classification, picture description and chart are understood as "enrichment," and whether or not downstream is really needed. In the formula that I've been working on, in the dense and complex layout sample, MinerU is often more stable, so I'm used to letting it take over this part, and then map it back and do it alone. This is my sample experience, not the uniform ranking of the two items on all the files.

MinenerU: Heavy OCR system

MinernuOfficial online servicesThe site is also available at the end of the page and at the API, saving local download models and preparing for display. Prices are relatively friendly, but I'm still used to local deployment when I have a card.

API model_version There are three values:

model_version	Apply Input	Characteristics
pipeline(Default)	PDF, Picture, Office	Traditional module resolution, speed and certainty; digital primary PDF, first batch screening priority
vlm(Officer's web label “Recommended”)	PDF, Picture, Office	Complex layouts, scanned copies, formulae and table-intensive pages are more worthwhile to try; higher costs are also needed to prevent the creation of errors
MinerU-HTML	HTML	HTML Text Extracting Special Mode; not ratio vlm Do not select a higher-level, non-HTML file

function parameter,is_ocr Default false，enable_formula and enable_table Default true，language Default ch; also available page_ranges Only sample pages are parsed. Formula Switches on vlm Down only affects the intra-line formulae. It's usually Docling who's in the MinerU, which is a non-processed scan or complex document, so I'm in MinerU more often. Choose vlm;digital, generic PDF still to be tried first pipelineThere is no need to spend VLM more than once for the sake of uniformity.

Default for the current open-source version hybrid-engineI'm sorry. It combines primary text extraction with VLM, with the goal of reducing hallucinations while retaining high accuracy. It has. medium and high 2-Back: Default medium Faster, but not supported; maximum precision or photo analysis required highI'm sorry. And... pipeline、vlm-engine Two local models.pipeline It can be run on CPU or GPU, emphasizing stability and non-supplegicity;vlm-engine In exchange for precision in complex layouts.

How to design a unified data expression layer

After the diversion, two solvers produce two sets of formats, and more will be introduced later. It requires a unified expression before the inquiry can be made on it.

The question that this layer is going to answer is really certain: what are the elements in the document, what type each element is, who is it, who is it, who is it read, what order is it, what is it from, what coordinates are to explain, how the row relationship of the table is, how the model is created, how the content is distinguished from the original text, and whether the original elements can be recovered after cutting off the chunk.

The DoclingDocument, I think, is the better designed one in this kind of structure, and it has clear fields for each of the questions:

Means questions to answer at the level	DoclingDocument
What are the elements?	texts / tables / pictures / key_value_items The stylized packagings are separated; the new version also includes form_items、field_regions、field_items
What kind of elements are they?	DocItemLabel Enumeration, 30+ labels, from SECTION_HEADER Present. FOOTNOTE
Who's got who?	body For root trees, use JSON pointer (e.g. #/texts/1I'm a father and a son.
Reading Order	Tree depth priority through the field without additional order fields
Where did it come from?	ProvenanceItem- Yeah. page_no、bbox、charspan
What do you mean, coordinates?	BoundingBox Take your own CoordOrigin Enumeration, left top or left bottom in the data.
Table Structure	TableItem.data It's structured. TableCell, Ham row_span、col_span、column_header
Content generated by the model	It's in its place. meta barematic fields, separate from original text fields; old versions of pictures, tables annotations Disabled
chunk, how do you trace it?	BaseChunk.doc_items Point back to the original elements that make it.

There are a few points worth saying alone.

Trees, not lists. Reading sequences and hierarchy are the same thing in trees, and they are sequenced over and over and over and over and over and over and over and over and over and over again, and you know the chapter path by looking at the parent. A flat list with a whole number can be expressed, but it has to be pushed again every time you need it.

The coordinates are written in the data.CoordOrigin It is an itemization rather than an agreement, because the PDF primary coordinates are the bottom left, the image processing practice is the top left, and neither of the two sides can be connected by default. Of course, most of the search scenes do not need to be precise enough to be a frame, and this field is not a bad idea, but when it is really a good one, it saves a kind of hard-to-see fault.

Tables are structured cells rather than HTML strings. Answer."What's the column title for this row?"The former read the field directly, while the latter re-deciphered HTML.

Original language and creation of content subfields. Current schema with item meta Saves derived information such as photographic descriptions, classifications and table summaries, and the text remains in the content field. Old pages of pictures and tables annotations Compatible accesses are also maintained but marked as obsolete. This corresponds to the one in front that "can't make the model's story look like a document."

Key premise docling-core It can be installed independently, without dependence. docling Main package, pure Pydantic model, officially stated as being"Interoperability"Designed. In other words, it was intended to be used as a generic representation by third parties, not as an internal realization detail for Docling.

And it's bringing more than schema.HybridChunker The hierarchy and tokenization-aware cut-off also combines small pieces under the same heading, with the output chunk taking headings, captions and position information; sequenced API for Markdown, HTML; Langchain, LlamaIndex, Haystack, CrewAI. Use DoclingDocument as a semantic expression layer to reduce the cost of subsequent adaptation work.

The rest is about the adapter that MinerU output to DoclingDocument. There's no such thing as a smart spot, but there's a lot of little engineering. I'm not gonna let you go, Legacy. content_list.json It's a reading-order list. The title level is on it. text_level (a) Revert; 3.0 from all backends also output grouping by page and uniforming to type + content It's... content_list_v2.jsonThe new adapter is more suitable for priority, but the official V2 is still marked as a development format. And the coordinates can't be mixed:content_list.json And V2 bbox The map is 0-1000.middle.json Use corresponding page_size . The page size coordinates system, VLM Original model.json Use 0-1. Plus, MinerU. page_idx From 0, Docling page_no From 1 onwards, and the conversion of HTML tables to a cell grid with spans, such errors are usually not reported wrong, but only allow the reference to be slushly slurred over a page.

RAGFlow also walked this way. It's in... v0.22.0 Medium experimental to add MinerU as an optional PDF solver, not using the Docling expression, but rather suitable for a section structure that fits into a home. The output particle size of the external solver and the internal chunk structure still need to be aligned, and alignment is not as easy as it could be expected.

Show layer is not query layer

With a uniform indication, it seems logical to search the matter. But there's a lot of confusion here.

DoclingDocument."What's the file like?", don't solve"How do I find it?"I'm sorry. It's a tree. Trees are good at placing and sequencing. They're not good at answering."Which documents refer to X and are published after 2025"I'm sorry. These two things require different data structures, and Docling himself has opened them up in different warehouses.

I prefer three: the most traditional keyword queries, PageIndex, DeepRead, which allow models to navigate themselves on document structures, and intensive vector-drive search. That kind of knowledge mapping technique that I used to choose. In the common knowledge base, the physical type is easily driftable and the cost of extraction, discrimination and updating is much higher than that of tree navigation.

Structured queries are indexing the tree into a relationship table or search engine, filtering by element type, chapter path, page number, document ID, and walking BM25 in the body. The most reliable, precise and interpretable answer to the question of clear filtering conditions is the most reliable and reliable.

PageIndex, DeepRead, such techniques are searchable like human beings: first look at the stratification catalogue, then look at the chapter title, move down step by step, and finally find what is needed. PageIndex simply calls itself a vctorless, and the whole process does not need a vector bank to allow the model to determine which branch to go on a tree by node; DeepRead makes it a loop of locate-then-read, and locates it and reads it. Their common advantage is that each step can be explained and the result can be directly on the page and chapter.

Vector search is by itself. HybridChunker The output chunk, which carries its own ancestral titles and annotated drawings, puts this information into embedded text and quantifys it, and it is much better to recall semantics. The chunk retains the reference to the original element, and can go back to the structure and the original page after the hit.

These are not competitions, but different views from the same factual level. Integration should take place at the retrieval level, where each is called back and reordered, rather than forcing a universal index at the resolution level. The function of the typology layer is to keep the facts right, to keep them intact and to be able to trace them back to their original location at any time.

But all these callers have a common premise: the document must have a clearly structured tree. The catalogue is correct, the chapter level is not disordered, the title and the body are attributed to the right, and it is directly determined whether the model can be navigable. That's why I'm willing to show you how much energy there is, not just to save it for good, but to really rely on it on the back floor.

Concluding remarks

The whole chain is actually four levels: identification, structure, representation, searching.

The identification is only a small first paragraph, which is followed by the restoration of the page to a structural object, and the consolidation of the output of multiple solvers into a single single expression, with the final turn of query. It means that the layers are designed to determine what can be done directly behind them, but it means that doing it better is not the same as being able to find out.

So the title, and the article, the story is twisted. OCR cannot fit the current name, and now the link is about structural restoration, unity, retroactivity and retrieval interface. But this does not prevent it from being much more important than it was in the past, and this is the first and the easiest to underestimate for Agent, who needs to process real documents.

MOGA。

References

• Docling, DoclingDocument concept
• Docling, Chunking
• Docling, Serialization
• Docling, Enrichments: picture description
• Docling, Document converter API (SimplePipeline / StandardPdfPipeline)
• Docling, Advanced options
• Docling, Vision models
• Docling, REST API
• Docling, Managed service
• IBM, Docling for IBM watsonx
• docling-project, docling-core
• OpenDataLab, MinerU
• MinerU, Official online service
• MinerU, Online API documentation
• MinerU, CLI tools
• MinerU, Output File Format
• MinerU, Changelog
• OpenDataLab, MinerU2.5-Pro: Pushing the Limits of Data-Centric Document Parsing at Scale
• RAGFlow, v0.22.0 release notes
• Baidu, Unlimited-OCR
• Baidu, Unlimited OCR Works: Welcome the Era of One-shot Long-horizon Parsing
• vLLM Recipes, baidu/Unlimited-OCR
• VectifyAI, PageIndex
• DeepRead
• Sarthi et al., RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval
• Google Cloud, Document AI Layout Parser
• Microsoft Azure, Document Content Understanding: Markdown Representation
• Amazon Web Services, Tables in Amazon Textract