Skip to content
All articles
Audit practice 8 min read· by Lars Zimmermann

AI visual inspection: what “95 per cent accuracy” leaves out

An AI visual inspection that waves every part through is 99% accurate at a 1% defect rate. Which two figures belong in acceptance and who sets the threshold.

In short

In AI visual inspection, an accuracy figure without a reference says almost nothing. At a defect rate of 1 per cent, a system that waves every part through is 99 per cent accurate and finds no defect. Acceptance needs two figures, escapes and false rejects, plus a test with unknown defect types and a person who signs off the threshold.

Auf Deutsch lesen: deutsche Fassung

A practitioner from industrial AI explains in an interview what customers demand of an AI visual inspection: more than 95 per cent accuracy. That sounds like a clear requirement. Which accuracy is meant, he does not say. And that is exactly where the problem begins.

I run a precision engineering company myself. With us, an inspection decides in the end whether a part leaves the building. Anyone who offers me an inspection that is 95 per cent accurate gets a question back from me: 95 per cent of what?

The answer is no quibble. The same figure can be used to sell a system that finds every defect and one that finds not a single one. I do the sums for you below. After that come the two figures that belong in every acceptance, and the two questions that are almost always missing.

The system that does nothing is 99 per cent accurate

Take a line with 10,000 parts and a defect rate of 1 per cent. That is 100 defective parts and 9,900 good parts. Now two inspection systems.

  • System A waves every part through. It is right for 9,900 of 10,000 parts, that is, 99.0 per cent accuracy. It has not found a single defect.
  • System B detects 95 of the 100 defective parts. In return it rejects 2 per cent of the good parts, that is, 198 parts. It is right for 95 plus 9,702 parts, which is 97.97 per cent.

By this calculation the useful system is less accurate than the useless one. Anyone who measures only accuracy across all parts in acceptance will, in case of doubt, accept system A. It passes any test set at 95 per cent and keeps its promise. It just does not inspect anything.

The reason is simple. Defective parts are rare. A metric that counts all parts equally is dominated by the good parts. The better your process runs, the less it says. At a defect rate of 0.1 per cent, system A reaches 99.9 per cent and still finds nothing.

And “accuracy” is not the only word that can be bent. In a quotation it can also mean the detection rate, that is, the share of defective parts the system finds. Or the share of real defects among everything it rejects. Three different quantities, one word. If the requirements specification says only “95 per cent accuracy”, the vendor may pick the most favourable one.

The figure is not entirely without value, though. If the vendor means the detection rate by 95 per cent, measured per defect type on parts from your production, that is a strong commitment. Then only the second half is missing, the price for it in good parts. So do not ask whether the figure is correct. Ask what it refers to and what it was measured on. A reputable vendor answers that in one sentence and adds the confusion matrix: how many defective parts detected, how many let through, how many good parts let through, how many rejected. Any percentage can be recalculated from these four numbers. Without them, none can be checked.

Two figures belong in acceptance

Quality assurance has had two terms for this for decades. They fit any automatic inspection, with AI or without.

  • Escapes: defective parts that slip through the inspection. With system B it is 5 out of 100. Escapes end up at the customer, as a complaint, as an 8D report, in the worst case as a recall.
  • False rejects: good parts that are rejected as defective. With system B it is 198. False rejects stay in the building, as manual re-inspection, as rework or as scrap.

Both figures have a price, and it is never the same one. For a visible part with cosmetic requirements, false rejects are expensive and escapes are annoying. For a part that goes into a brake or a steering system, escapes are the only thing that counts. That is why there is no generally good threshold. There is only one that fits your part, your customer and your re-inspection capacity.

A vendor who has understood this gives you both figures without being asked, measured on your parts. A figure from the lab, measured on a public data set with scratches on sheet metal, says nothing about your turned part. Ask for the number of defective parts in the test data set. If the test contains twenty defective parts, any percentage given to two decimal places is false precision. A single missed part then shifts the detection rate by five points.

The same 98 per cent, ten times as much re-inspection

The second trap only shows itself in operation. Suppose your process improves and the defect rate falls from 1 to 0.1 per cent. System B stays as it is: 95 per cent of defective parts detected, 2 per cent of good parts rejected.

  • For 100,000 parts that is 100 defective parts. The system finds 95, 5 get through.
  • Of 99,900 good parts it rejects 1,998.
  • The re-inspection bin holds 2,093 parts. 95 of them are really defective. A good 95 per cent of the bin are good parts.
  • Accuracy across all parts is 98.0 per cent, almost exactly as before.

At a defect rate of 1 per cent, just under a third of the rejected parts were still real defects. Now it is one in twenty-two. The metric in the acceptance record has not moved. The operator at re-inspection experiences a different system. All day long he sees good parts that are supposedly bad, and he starts waving the bin through. Then you have escapes that no record shows.

That is why a third item belongs in acceptance, one that follows from the first two: how many parts per shift end up in re-inspection, at your current defect rate and at the one you expect in a year? That is a staffing question, not statistics. Anyone who does this calculation before buying gets no surprise after buying.

What the model has never seen

A model for visual inspection learns from examples. It knows the defect types that occurred in the training data set: scratches, blowholes, burrs, missing chamfers. What happens to the crack that never occurred in training because it only appears every two years? A model that only distinguishes between known classes assigns it to one of them, in the unfavourable case “good”.

There are systems that learn differently. They are shown only good parts and report everything that deviates from them. An unknown crack is more likely to be noticed that way, because it does not look like a good part. The price is usually more false rejects, because harmless deviations are noticed too, such as a spot of cutting fluid. Which approach fits better depends on the part. What matters is that you know which one you are buying, and that the vendor says how its system deals with the unknown.

An acceptance should test this, with parts that were not shown to the model beforehand. Hold back a set of real defective parts, including defect types missing from training, and only let the system loose on them at acceptance. That is the idea behind ISO/IEC 42001, Annex A.7.4 (quality of data for AI systems): define requirements for the data and check whether the data meet them. For visual inspection this means, in very practical terms: enough defective parts per defect type, photos under the lighting conditions of your shop floor, parts from more than one batch.

It does not end with acceptance. Tools wear, a new raw material supplier delivers a different surface, someone changes the lighting. Each of these changes can shift the detection rate without the system issuing any message. The only way to notice is a regular manual sample from the parts the system has let through as good. ISO/IEC 42001, Annex A.6.2.6 (AI system operation and monitoring) requires performance to be monitored in operation. In visual inspection, this sample is the evidence of that.

Who signs off the threshold?

Every AI inspection system has a threshold. The model outputs a value for each part, and from a certain value the part counts as defective. Lower the threshold and escapes fall while false rejects rise. Raise it and the reverse happens. There is no setting that lowers both.

That makes the threshold a decision about risk and cost. It should be taken by the head of quality, with date and reasons, not by the vendor’s commissioning engineer on the last evening before the start of series production. In many companies, however, it is recorded nowhere. It sits in a configuration file, and whoever changes it changes the inspection process without anyone noticing. Record the threshold, the reasons and the approval, like any other change to the inspection plan. That is the core of ISO/IEC 42001, Annex A.6.2.3 (documentation of AI system design and development): decisions that determine the behaviour of the system should be documented, with the reasons.

And what does the law say? For most visual inspections in manufacturing, little. Since the Digital Omnibus, Regulation (EU) 2026/1744, an AI system used exclusively for non-safety-related aspects of quality control is not considered a safety component (Article 6(1a) of the EU AI Act). The exemption has a reverse side: if a failure of the system would endanger health and safety, it is considered a safety component after all (Article 6(1b)). Anyone inspecting parts for brakes or steering should not leave the question open.

It is interesting how the legislator describes accuracy for high-risk systems. Article 15(3) of the EU AI Act requires the instructions for use to declare the levels of accuracy and the relevant accuracy metrics. Metrics, in the plural. Article 13(3)(b)(ii) additionally requires the known circumstances that may affect the expected level of accuracy. For your visual inspection this is not an obligation. It still serves as a benchmark for an honest requirements specification.

Five points I would write into every requirements specification for an AI visual inspection:

  • Escapes and false rejects as separate target values, per defect type, measured on your parts.
  • The number of defective parts in the acceptance test, with at least one defect type that did not occur in training.
  • The expected volume in re-inspection per shift, at the current and at the targeted defect rate.
  • The threshold as a documented value, with name, date and reasons, and a rule on who may change it.
  • A manual sample from the parts let through, with frequency and a limit beyond which acceptance is repeated.

With these five points, a quotation with “95 per cent accuracy” quickly loses its magic, or it passes. Both are good news. One question remains that only you can answer: who in your company would notice today if your inspection system had been letting every second crack through for three weeks?

Share: LinkedIn E-Mail

Frequently asked questions

What is the difference between escapes and false rejects?+

Escapes are defective parts that the inspection lets through as good. They end up at the customer. False rejects are good parts that the inspection rejects as defective. They cause re-inspection, rework or scrap in your own company. An automatic inspection needs a separate target value for each.

Why is 95 per cent accuracy not enough as an acceptance criterion?+

Because defective parts are rare. At a defect rate of 1 per cent, a system that waves every part through is 99 per cent accurate. The figure is dominated by the good parts and does not say how many defects the system finds. On top of that, “accuracy” can mean three different metrics.

Is an AI visual inspection a high-risk AI system?+

As a rule, no. Since the amendment by Regulation (EU) 2026/1744, an AI system used exclusively for non-safety-related aspects of quality control is not considered a safety component (Article 6(1a) of the EU AI Act). If its failure would endanger health and safety, however, it is considered a safety component after all under paragraph 1b.

Who should set the threshold of an AI inspection?+

The head of quality, with date and reasons. The threshold trades escapes against false rejects and is therefore a decision about risk and cost. It should be documented like any change to the inspection plan, with a rule on who may change it.

Author & expert review: Lars Zimmermann · ISO/IEC 42001 Senior Lead Auditor & Senior Lead Implementer · ISO/IEC 27001 Lead Auditor & Lead Implementer (PECB)

Last updated: 10 November 2026. Researched and reviewed to the best of our knowledge; not a substitute for individual legal advice.

Questions about your own case?

In a free 15-minute intro call we assess where you stand on ISO 42001, ISO 27001 and the EU AI Act, honestly and without a sales pitch.

Continue reading