How Does Object Detection Work?

Detection is not classification

Start from what an image classifier does. You hand it a picture, it looks at the whole image, and it returns a single answer: this is a cat, or a beach, or a license plate. One image goes in, one label comes out. Even when it shows you the top five possibilities, all five are competing to describe the entire picture. The classifier is being asked a question about the image as a whole.

A detector is being asked something different: for every object in this picture, what is it and where is it. The output is not one label but a list of detections, and each entry pairs a class with a location — usually a rectangle — and a score. That is why detection can return zero entries for a clear sky, one for a single dog, and a dozen for a busy street. Detection does not just name the objects; it points at them.

That extra “where” is the whole difficulty. Classification can throw away layout: resize the image, ignore the edges, care only about overall content. Detection cannot. A detector has to answer “what” and “where” together, and it has to do it for an unknown number of objects that might overlap, sit at any scale, or appear at any position. Everything else in this guide — the two families, the redundant boxes, the thresholds, the failure modes — is a consequence of demanding a location as well as a name.

One point to hold onto, because it explains so many surprises later: a detector's output is attached to objects, not to categories. A classifier always says something, because “what is this picture” always has an answer. A detector can honestly say nothing, because “where are the dogs” may have no locations to return. Detection fails by silence. Classification fails by being wrong.

• Classifier output: one label for the whole image.
• Detector output: a list of entries, each a class, a box, and a score.
• The two share feature extraction and differ almost entirely in how the location gets produced.

The output is a box, and the box is the problem

When a guide says “bounding box,” it means the smallest axis-aligned rectangle drawn around an object — top edge, left edge, width, height, no rotation. It is the cheapest location representation that is still useful, and almost every detector produces it, including the browser tool linked at the end of this guide.

Concretely, one detection is a handful of numbers. The label is the class the model thinks it found. The box is stored as four numbers — either the top-left corner plus width and height, or the center plus width and height. The score is a single number, typically between 0 and 1. That is the entire data structure, and every detector exists to output these numbers.

The box shape carries a hidden assumption: that an object can be enclosed tightly by a rectangle that does not tilt. When that assumption breaks, so does the representation. A cat curled into a round blob wastes most of its box on background; a long, thin, diagonal object is almost all empty space. And because a detector predicts only a rectangle, it cannot tell you where the object actually ends inside it. That is why “where” is harder than “what” — you commit to one geometric shape for a position before you have found the object — and it is the precise reason a detector is not a segmenter.

The two families of detectors

Every detector has to decide what and where at once. Two strategies dominate the field, and they trade speed against accuracy in opposite directions. The difference is not quality — it is where the location question gets answered.

Two-stage detectors: find regions, then label them

A two-stage detector splits the job. Its first stage proposes places worth looking at: a set of candidate regions that might contain something. Its second stage takes each candidate in turn, decides whether it actually holds an object, which class, and nudges the box to fit better. The famous members of this family — R-CNN and its descendants, Fast and Faster R-CNN, Feature Pyramid Networks — differ mainly in how those candidate proposals get generated, from an external proposal method to a learned proposal network living inside the backbone.

Because the second stage examines a handful of promising candidates at full attention, two-stage detectors tend to be accurate, and they have historically done better on small or hard-to-distinguish objects. The cost is time: looking at every candidate twice is slow per image. The strategy is also easy to teach — make guesses about locations, then check each guess properly.

Single-stage detectors: predict boxes and classes in one pass

A single-stage detector does not separate location from classification. It treats detection as dense prediction: run the image through a backbone once, and at each output location directly predict a small set of boxes and a distribution over classes. There is no “wait, let me look at that region more closely” step — everything comes out of one forward pass.

That makes them considerably faster than two-stage detectors, which is why they dominate real-time work, and it is what YOLO means — You Only Look Once, one pass over the whole image. The idea is easiest to state: divide the feature map into a grid of cells, and let each cell propose boxes for objects whose center falls inside it, with a class score for each box. A detector trained this way names and locates every object in a single shot, with no proposal stage anywhere. Because it is never asked to reconsider a location, the single-stage approach has historically been weaker on tiny objects, where the pixels per object barely encode anything — a gap modern variants close by putting more effort into the small-object end of training. The trade-off remains: single-stage for speed and simplicity, two-stage for careful localisation on hard cases.

A third pattern, in plain terms

Transformer detectors that appeared more recently — including the DETR-style model the linked browser tool runs — break the two-stage / single-stage frame in a useful way. Instead of generating dense boxes at grid locations and filtering them afterward, a small set of learned queries each attend to the whole image and output a final box directly. Because the model was trained not to predict the same object twice, this style generally needs no separate duplicate-filtering step. You still choose thresholds and live with a fixed label set, but proposal and de-duplication come from one trained system rather than being stitched together at the end.

Why one object gets many boxes, and how they get merged

Run a detector on a picture of a single bicycle and you might get six boxes that all sit on it, all labelled “bicycle,” with slightly different edges and different scores. That is not a bug. A detector does not decide where an object is once; it proposes many candidate boxes across the image and scores each, and an object under enough of those candidates attracts a crowd of overlapping hits — every grid cell that contains it, every anchor shape that roughly fits, every feature location whose receptive field covers it. The system produces redundant answers on purpose: it was built to be exhaustive, and missing the object entirely is worse than naming it twice.

Non-maximum suppression, or NMS, is the cleanup. Applied to a pile of labelled boxes it works like this: sort every box by score, take the highest-scoring one and add it to the final list, then throw away any remaining box of the same class that overlaps it too much — measured by intersection over union, the fraction of the two boxes’ area they share. Repeat: the next-highest surviving box becomes the new keeper and its near-duplicates get pruned, until nothing is left. What survives is one box per object, and it is whichever candidate the model liked best.

The threshold that decides “too much overlap” is a tuning knob with visible consequences. Set it aggressively and adjacent objects of the same class merge into a single detection — two cars side by side come back as one wide box, because the second was pruned as a “duplicate” of the first. Set it loosely and each object keeps several overlapping boxes. NMS is a small piece of logic and one of the most common sources of strange output when it is misconfigured. One caveat, since it comes up with the linked tool: NMS is a rule the classic detector families bolt on after prediction; a DETR-style detector is generally described as replacing that post-hoc filtering with a learned assignment that avoids emitting the duplicates at all.

What a confidence score is — and what it is not

Every detection carries a score, usually written as a number between 0 and 1 and shown to you as a percentage. It is worth being blunt about what that number is: a learned value the model was trained to emit, high when its features look like a member of this class and low when they do not. It is a signal produced by the network, not a physical measurement, and it is not calibrated to mean anything you can use outside the model that emitted it.

“Not a probability” is the part people get wrong. A box scored 0.90 does not mean “there is a 90% chance this is a dog.” A well-behaved score correlates with reality — high numbers usually go with real objects, low numbers with noise — but correlation is not calibration. The same model's 0.90 means something different for “person” than for “toothbrush,” because the classes have different appearance ranges and different numbers of training examples. What a score is legitimately good for is comparison within one model on one run: this box is more likely a real dog than that one, this label beats that label for these pixels.

Because scores are relative, the practical question is where to cut. Every detector surfaces a threshold: keep detections whose score is at or above it, discard the rest. Say a box scores 0.58 — it survives when the threshold is set to 0.50 and disappears when the threshold is raised to 0.60. Nothing about the image changed, and nothing about the model's opinion of the pixels changed either; only the reporting rule moved. So the same picture legitimately produces different output lists at different thresholds, and that is not the model being inconsistent. It is you deciding how much uncertainty you are willing to accept.

Thresholds are where detection meets real cost. A system that misses an event is often worse than one that flags a false one, so you set the threshold low and clean up noise later; a line that halts for a phantom defect costs money, so you set it high and risk a rare miss. There is no “right” number inside the model — confidence is a dial, and the honest use is to decide up front which side of that trade-off hurts you less.

The label set is fixed, and it cannot say “unknown”

A detector outputs only classes from a list decided during training. The canonical example is COCO, a benchmark whose label set covers 80 everyday categories — people, cars, animals, common household items — and most “general” detectors you meet, including the browser tool at the end of this guide, ship exactly that list. You cannot ask such a model about a category it was never trained to recognize; it has no slot to put the answer in.

What happens when you show it something off the list is not a polite refusal. There is no “unknown” output. A detector can only do two things: assign one of its known labels, or return nothing for that region. So an object outside the label set gets absorbed into whichever known category looks closest — a robot vacuum comes back as “cat,” a shopping cart as “chair,” a ride-on toy as “car.” Nothing crashed and nothing warned you; the detector reached for the nearest label it had and used it with a straight face. That is a failure mode with no error signal, which makes it dangerous in a way the missing-object case is not.

The flip side is just as misleading. An object that is on the list but rendered strangely — an unfamiliar angle, a rare colour, a shape the training set underrepresented — can score low enough on every known label that it disappears entirely. The detector's silence does not mean “there is nothing here”; it means “nothing matched any label I know well enough to report.” The 80 categories are a fixed vocabulary, and anything outside it is either paraphrased into something close or left unspoken.

One consequence to spell out, because it comes up constantly: a label is a category, not an identity. A detector that boxes a figure and writes “person” has not recognized anyone — it has asserted that the pixels look like the class “person.” Whether that person is someone you know is a separate model built on top, and one with far heavier consequences when it is wrong.

Where detection struggles

Once you know how a detector forms its answer, its weaknesses stop being mysterious. Each of the following breaks a specific assumption baked into the mechanism, which is why each fails the way it does.

Occlusion and unusual viewpoints

A detector learns what an object looks like when most of it is visible and in a typical pose. When the visible part shrinks — a car half-hidden behind a pillar, a face turned sideways, a person seen only from the shoulders — the evidence in the image no longer resembles the training examples. Unusual angles are the same problem: a cup viewed from directly above has an oval rim and no handle, a laptop from the side is a thin wedge. The model saw mostly front-on, well-framed objects, so it is brittle once you change the camera's relationship to the thing it is looking at.

Tiny objects and crowded scenes

Small objects carry a brutal constraint: a distant bird or a face across a street occupies a handful of pixels, and after the image passes through a real network's feature pyramid, very little of that survives. Detectors routinely miss things that are perfectly visible to you, especially when the input was downscaled — exactly why the linked tool warns that small objects go missing. Crowds are the opposite failure with the same root: when many same-class objects sit close, their candidate boxes overlap heavily and NMS prunes the survivors as duplicates of each other, collapsing a dozen pedestrians into two boxes. Fix the first only with more input resolution; fix the second by loosening the overlap threshold and accepting more false positives.

Reflections and resolution mismatch

A reflection can produce a real, well-shaped “car” in the pixels with no physical object behind it — the detector finds the appearance and reports it, because its job is to find appearances. Mirrored glass, a shiny surface and a poster of a dog all trigger this. The subtler case is resolution mismatch: every detector was trained at a range of input sizes, and feeding it an image far from that range — a huge photo shrunk drastically, a thumbnail blown up — changes the pixel footprint of every object. A model that learned “a dog is roughly this many pixels wide” stops noticing a dog at a tenth that span. This is the same mechanism that makes downscaling risky, and why detectors often need a matching preprocessing step to behave at all.

Where detection actually gets used

Detection earns its place wherever the question is “is that thing there, and where” rather than “what is this image.” The categories below are stated generally: they describe what the mechanism is good for, not any one product's marketing.

Retail and inventory: counting stock on a shelf, checking a display against a plan, flagging gaps where a product should be, spotting misplaced items. A camera points at the shelf and a model finds each product, so you can count and locate them rather than merely classify the shelf as “mostly full.”

Packaging and quality inspection: a missing cap on a bottle, an object on the wrong conveyor, a part placed upside down. The pass-or-fail question is about specific items in specific positions, and a classifier forced to describe the whole image cannot answer it.

Traffic and counting: detecting vehicles, measuring flow from a fixed camera, counting pedestrians, spotting queues at an intersection. Detection gives the per-object positions; a separate tracking step is usually added when you need the same object followed across frames rather than counted once.

Robotics and assistance: a robot arm needs the object's position before it can grasp, and a navigation aid for a visually impaired person needs to say “doorway ahead.” In both the value is in the box, not the label. Every one of these applications shares a hard requirement: the label set has to actually contain what you care about, for the reason in the previous section.

A detection demo you can run in your browser

The page linked from this guide runs the mechanism locally. You give it one image — PNG, JPG or WebP — and it downloads the model weights on first use, then reuses them from your browser's cache afterwards. Nothing about your photo is uploaded; the pixels are read, resized and passed through the detector inside the tab, and the annotated image is drawn on a canvas in the page.

Under the hood it runs a DETR-style transformer detector over the 80 everyday COCO categories, using a quantised copy of the weights small enough to ship to a browser. It returns a detection for each object it finds: a bounding box drawn on the image and a confidence score in the results table, and you can download the annotated image. It filters what it shows by the same kind of score threshold described earlier, so raising it drops low-scoring boxes and lowering it adds them back.

It is worth being explicit about the limits, because “object detection in your browser” invites false expectations. It handles one image at a time. It downscales your image before inference, which is exactly the resolution mismatch discussed above and the reason small objects can simply not show up. Its label set is fixed at those 80 COCO categories, so it cannot name anything outside them. It does not identify people: a box labelled “person” is a category, not recognition of anyone. There is no segmentation, no mask, and no video tracking, and there is no counting feature beyond the number of rows in the results table. Treat it for what it is — a working demonstration of boxes, scores, thresholds and the single-pass mechanism, running on your own copy of the model — not a production detector, and it will not measure its own accuracy.

Frequently asked questions