How Does OCR Work? The Pipeline, Stage by Stage

What OCR actually is

OCR answers a deceptively simple question: given an image that contains text, what does the text say? The input is pixels; the output is characters. There is no direct mapping between the two — a capital A in one font, at one size, on one printer, is a different pixel pattern from a capital A anywhere else in the world. Everything an OCR engine does exists to bridge that gap.

Real engines do not recognize individual characters in isolation the way people imagine. They narrow hypotheses: which parts of the image are text at all, how the text is arranged into lines and blocks, and then which string of characters is most plausible given this line image and this language. Recognition is a search over plausible readings, not a lookup of letter shapes.

A distinction worth making immediately, because “how does OCR scanning work” is part of this query: OCR is not scanning. A scanner or camera produces the image; OCR reads it. “Scan to text” means both steps run together, and the quality of the first sets a hard ceiling on the second — no engine can reconstruct ink your device never captured.

• Input: a raster image of text — a scan, a photo, or a screenshot.
• Output: the reconstructed characters — plain text in simple tools, or a positioned text layer inside a PDF product.
• Every failure discussed below traces back to one stage of the pipeline in the next section.

The four stages of the OCR pipeline

Every OCR engine, from the batch systems of the 1970s to the LSTM model running in a browser tab today, does the same four jobs in roughly this order. Knowing which stage owns which failure tells you what you can fix.

Pre-processing: making the image readable

Pre-processing prepares the image before any character is looked at: grayscale, binarization to black and white (usually adaptive thresholding, which handles shadowed and bright regions differently so faint areas survive), a few degrees of deskew, sometimes despeckling or sharpening. Almost every step throws information away on purpose, because downstream stages work on a simplified picture.

Its failures are the failures you bring to it. Severe skew can defeat the deskew estimate itself, especially on short text. Low contrast — gray text on gray ground, a faint pencil original — collapses during binarization, and characters get swallowed as if never printed. JPEG artifacts ring small glyphs with halos of false-color pixels that later stages read as stray marks or fuse into the letter. And very small type loses its distinguishing features first: the hole in an “o”, the crossing on a “t”, the ascender on a “d” — exactly what separates them from their neighbors.

These are also the failures you can fix at the source: more light, more resolution, a lossless save, a flat page.

Layout analysis and segmentation: finding the text and its order

Before recognizing anything, the engine has to find where text is and rule out where it is not — photographs, rules, borders, stamps. Then it cuts text regions into lines, and lines into word candidates. Layout analysis is also where reading order gets decided.

Failures here are not wrong letters but wrong structure. A two-column page analyzed as one wide column interleaves the texts line by line and produces output whose every word is individually fine and whose overall meaning is nonsense. Tables with faint borders get cut into columns at unpredictable places, letter-spaced headings split into several “words”, and superscript footnote markers wander into the middle of the sentence they sit above. Because every later stage trusts this one completely, a layout error cannot be undone downstream.

Recognition: turning line images into characters

Recognition converts the image content of a text line into characters. Classic engines did this one glyph at a time, comparing each clipped-out letter shape against learned models. Modern engines do it a line at a time: the whole line image goes into a neural network and a character sequence comes out. In both eras the engine proposes, and its knowledge of what is plausible — a word list, a language model — does much of the deciding.

Its signature failures are confusions between visually similar things: zero against capital O, one against lowercase l and capital I, five against S, eight against B, “rn” against m. They come in two flavors. The image may be genuinely ambiguous — a printed 8 whose bowl closed badly reads as a 3 to anyone. Or the image may be clean and the model simply never saw that shape: an exotic display font, decorative typography, or handwriting, which printed-text models were not trained on. The first flavor is fixed with a better scan; the second is not fixed at all.

Post-processing: sanity-checking the result

Post-processing checks raw recognizer output against expectations: a dictionary of the language, rules for commonly confused pairs, patterns for numbers, dates, and currency amounts. Low-scoring words are the ones most likely to be revisited or flagged.

The mechanism that fixes “tlie” into “the” is the same mechanism that rewrites a correct product code into the nearest dictionary word. This is why OCR output can end up “cleaned” into something that was never printed — proper nouns, serial numbers, and addresses get normalized toward what the language expects. If that distinction matters to you, the fix is input quality and human verification, not the dictionary stage.

How OCR recognized text before neural networks

The pipeline above predates machine-learning OCR by decades; what changed is the recognition stage inside it. Character recognition went through three generations, and old quirks still show up in today's engines.

Template matching

The earliest approach binarized the image, cut out every connected blob of ink as a candidate character, and compared each blob against stored letter shapes, picking the closest match under scaling. It worked only when the font was expected: any change in weight, size, or print quality destroyed pixel-level similarity. Old OCR software being tied to a handful of typewriter fonts was template matching at work.

Feature extraction and classifiers

The next generation stopped comparing pictures and started comparing descriptions: does the glyph have a hole, how many strokes, do strokes cross, is there an ascender or descender. A classifier — a statistical rule, a decision tree, later a small neural network — mapped that feature list to a character. This survived font changes far better, but it lived and died on clean segmentation: two touching characters became one blob with the features of neither.

Tesseract's classic adaptive path

Tesseract historically ran both approaches: a traditional feature-based classifier first, then an adaptive classifier that, whenever it recognized a character confidently, added that shape to a template list for the current document. The engine literally learned the fonts of the page as it read them. That adaptive trick explains a legacy behavior that still surprises people: old OCR copes well with a strange font once it has seen the letters repeatedly, and badly on their first appearances — early pages of a document get worse output than later ones. When Tesseract rebuilt its engine around an LSTM, the classic path was kept as an option; the tool linked at the bottom of this page runs only the newer LSTM path.

What the LSTM era changed

The shift in modern OCR was to treat recognition as sequence prediction instead of character classification. A recurrent network — an LSTM, long short-term memory — can look at an image one vertical strip at a time and emit a stream of character predictions. A training and decoding technique called Connectionist Temporal Classification (CTC) lets the network learn from supervision that only says what the whole line reads, without labels for where each character's pixels start and end. Character segmentation stopped being a prerequisite.

That change removed the two biggest dependencies of classic OCR. Characters no longer need to be cleanly separated, because the network reads the whole line as one image. Fonts stop being a wall, because the network learns stroke-level visual regularities across enormous numbers of line images instead of memorizing per-character templates. On clean printed material, recognition largely stopped being a research problem; messy input became a training-data coverage issue.

It came with costs. A recognizer that never segments characters cannot tell you where each word is unless something else computes positions for it. Anything the model was not trained on — a script, exotic typography, a handwriting style — is now simply invisible to it rather than merely difficult. An LSTM engine's failure mode is often not a fuzzy answer; it is a confident wrong one.

How Tesseract's LSTM path works

Tesseract is the open-source engine that the linked browser tool uses (as tesseract.js, a port that runs it inside your tab). Its LSTM path processes a page in a specific order, and walking it makes the earlier sections concrete.

Page setup and layout

The image goes to grayscale and an adaptive binarization. The engine estimates the dominant text orientation, then classifies regions as text or non-text, detects the block and column structure, and cuts text blocks into individual text lines with bounding boxes. The order these boxes are sorted in is the order your text will come out.

Line recognition

Each detected line image is normalized — resized so the line has the fixed height the network was trained on — and fed into a convolutional feature extractor followed by a deep LSTM. The network outputs a probability-weighted stream of character predictions for the line, and CTC decoding collapses it into the single best reading. A compactly encoded language model participates during training and decoding, which is why the engine prefers plausible letter sequences over raw visual similarity.

Assembling the result

Word candidates receive confidence scores computed from the recognizer's own probabilities, then averaged for reporting. A dictionary-based pass repairs or flags doubtful words. Finally the text is concatenated in the reading order layout analysis picked — which is why a mixed-up column order cannot be repaired at this stage.

What the linked tool actually runs

The browser page starts the worker in LSTM-only mode — the engine's legacy pattern-matching paths are switched off — so this is exactly what happens to your file. The library's default worker code, engine core, and the traineddata model for whichever language you pick are fetched from a jsDelivr CDN. Exactly four languages are selectable: English, Spanish, Brazilian Portuguese, and Indonesian. Each model is roughly 2–15 MB, English around 10 MB; afterwards they sit in the browser cache, but that is a cache, not offline support: switching language recreates the worker and downloads an equivalent model, and a first run with no internet fails.

What confidence means, and what it cannot tell you

Confidence in OCR output is the recognizer's own estimate of how sure it is, normally reported as a percentage. Engines compute a score per word internally; the tool linked here shows a single average over the whole result. It is a useful signal, but only for one question: did the model see a shape that matched something familiar from training?

What it cannot tell you is nearly everything else. An average is a weak diagnostic — a smoothly reading, high-scoring page can hide one confidently wrong account number, because that number is a tiny fraction of the total. A model's probabilities are trained to be roughly calibrated on its training distribution, and documents are adversarial: a degraded 8 misread as a B is not less likely to be scored highly just because the ink was ambiguous. Sequence decoders also concentrate probability on whichever reading wins the competition, so a line that admits only one plausible interpretation inside the model can come back near-certain while being wrong about a name, a code, or a sign.

Read it as “the engine was in familiar territory,” not “this is verified.” Treat high average confidence as permission to spot-check, and treat numbers, identifiers, and names as if confidence did not exist — because for exactly the tokens where mistakes hurt, it effectively doesn't. The tool shows one average figure and exposes no word-level scores or pass/fail threshold, so it cannot tell you which word it doubted.

Where OCR goes wrong in practice

Setting aside letter confusions, a short list of input classes reliably defeats this pipeline. If any of them is your daily material, you already have intuitions about the output; here is the accounting.

• Multi-column reading order. Layout analysis usually handles two clean columns by reading one block fully, then the other. When the column gap is faint — narrow margins, a ruled table — the engine interleaves the columns line by line: every word individually fine, globally wrong order, and no post-processing stage can repair it.
• Merged digits. Where characters touch or nearly touch — dot-matrix printing, low resolution, tight kerning — a pair of ones can present as one wide glyph, and a line of figures loses its separators entirely. An LSTM line reader handles this better than classic per-glyph engines did, but digit strings in tables and forms are still where OCR quietly corrupts data.
• Ligatures. Older and high-quality typesetting joins several letter pairs into single glyphs; pre-nineteenth-century printing uses the long s, which looks like an f. Depending on what the model saw during training, you get the split letters, an exotic Unicode ligature codepoint, or a plausible-looking substitution.
• Stamps, postmarks and overprints. Rotated text, curved text, and text printed on top of other text defeat layout analysis before recognition even starts — the engine may classify the mark as a graphic and skip it, or try to read the postmark and the address as one pile of ink.
• Photographed pages with curl. Pre-processing estimates a single skew angle per page, and line normalization assumes every line shares one height scale. A curled page violates both mid-document: skew changes line to line, focus falls off toward the spine, and light gradients sweep across the sheet. This is why a flat scan beats a phone photo.
• Handwriting. Printed-text models — including the standard language data this tool downloads — were not trained for it. Handwriting recognition is a separate model class, and cursive remains the hardest common input in the field.

Adobe Acrobat, Bluebeam, and the text layer

Readers on this query often want to know how OCR works inside the document software they already own. Adobe Acrobat is a PDF product, and its OCR feature takes a PDF whose pages are pictures of pages, runs the pipeline above over each page image, and stores the result as an invisible, positioned layer of text behind the visible image. The page becomes selectable, searchable, and readable by screen readers. That output is commonly called a searchable PDF.

Bluebeam is a PDF markup product used heavily in construction and engineering. Its OCR option does the same job in that world: give scanned drawings and documents a text layer so their contents can be searched and copied. Its inputs skew toward large-format, rotated, text-sparse pages, which stresses the layout stages more than the recognizer.

The tool linked at the bottom of this guide is deliberately a different animal, and you should not read the two paragraphs above as describing it. It takes one PNG, JPG, or WebP image and outputs plain text — no layout, no word positions, no text layer, and no searchable PDF. If you need the invisible-layer behavior, you need a PDF product; if you need the characters themselves fast and without uploading anything, the browser tool is the fit.

What makes an image easy for OCR

Each tip below maps to a specific stage of the pipeline, which is why they work in a predictable order of value.

• Resolution first. Around 300 pixels per inch at the printed text size is the usual scanning target; small type wants more. If you cannot comfortably read a character at full zoom on screen, the recognizer cannot read it either — the ink detail the stages depend on is physically absent.
• Contrast without glare. Dark text on light ground, evenly lit. Flash hotspots and page-gutter shadows create local regions where binarization has no signal to keep.
• Flat and straight. A few degrees of skew is recoverable; a curl is not. Press the page down or feed it through a scanner.
• Choose lossless when you can. PNG for screenshots and document captures; one JPEG pass of a normal scan is fine, but repeated JPEG re-saving degrades small text before it degrades anything you notice.
• Keep the text area clean. Rules under lines, borders touching text, stamps across addresses, and handwritten annotations over print all push layout analysis into wrong decisions.
• Match the language setting to the text. The tool offers exactly four output languages, and picking the one the body text is in gives the language model its best chance; mixed-language pages degrade whichever choice you make.

Running the pipeline on your own image

The linked page runs the engine described above inside your browser tab with tesseract.js: pick a language, select a single PNG, JPG, or WebP file, and binarization, layout analysis, LSTM line recognition, and post-processing all happen locally. The recognized text appears with the character count and an average confidence figure, ready to copy or download as a .txt file.

The limits are the honest half of the description. Image files only — PDF, GIF, BMP and TIFF are rejected, so a scanned PDF has to become an image first by some other tool. Plain text out only: no preserved layout, no per-word bounding boxes, no searchable PDF. One file at a time.

The network behavior, once more because it trips people up: the first run downloads the worker code, the engine core, and the language model from a CDN — English is about 10 MB, the four models ranging roughly from 2 to 15 MB — and afterwards the browser caches them. So it is not an offline-first tool, and a first run without internet fails. Meanwhile the image itself never leaves the browser; the only traffic is that same public model file, fetched once by every visitor.

When something goes wrong, the page fails softly: an unsupported file type is rejected with an error, an image with no readable text gets a gentle no-text note rather than a crash, and if the engine itself errors the page releases the worker so your next attempt rebuilds it cleanly.

Frequently asked questions