Skip to content
All articles
Fundamentals 8 min read· by Lars Zimmermann

Automation takes the checkable work first, not the hard work

Difficulty does not decide which tasks go to AI first. Two axes, one documented measurement and what that means for the order of your projects.

In short

Difficulty hardly decides which tasks go to AI first. Two axes matter: whether the result can be checked against a yardstick and how long a person needs for the task. Short, checkable tasks come first, long ones are broken into steps with their own checkpoint. ISO/IEC 42001 A.6.2.4 requires defined criteria for verification and validation.

Auf Deutsch lesen: deutsche Fassung

The usual question is: what do we have that is complicated enough for AI to be worth it? It reliably leads astray. The order in which work is automated hardly depends on difficulty.

It depends on whether someone can check the result. And on how long a competent person needs for the task.

For both there is now a measurement instead of an opinion.

The measurement that makes the pattern visible

The research institute METR had AI agents and human experts work on the same tasks, under the same time budget. 61 experts, 71 runs of eight hours each. The models tested at the time were o1-preview and Claude 3.5 Sonnet. The result depends almost entirely on the time budget.

  • Two-hour budget: the best agents achieve four times the score of the experts.
  • Eight-hour budget: the humans are ahead again, though narrowly.
  • 32-hour budget: the humans achieve twice the score of the best agent.

So the curve turns. Over the short distance the agent is superior, over the long distance the human. METR states the reason soberly: humans currently get the better return from additional time. Agents get far quickly and then slow down. Experts start sluggishly and then pick up speed.

Added to this is a ratio that explains the direction: an agent generates and tests solution variants more than ten times as fast as a human, at a fraction of the cost. Where trying things out helps, it wins. Where you have to recognise that the whole approach is heading in the wrong direction, trying things out helps little.

This is easy to observe in a business. Have an agent write twenty variants of a product description and you get twenty usable variants in two minutes. Have it decide whether this product belongs in the range at all and you get a justification that sounds good and rests on nothing. The same machine, two completely different qualities of result. The difference lies in whether the question can be answered by trial and error.

Why checkability is the real threshold

The second part is more important than the numbers, and its starting point comes from METR itself, from a separate analysis of July 2025: benchmark tasks are cheap to run and can be scored automatically, and are therefore easy to improve on through reinforcement learning. My conclusion from this: anything that can be graded by machine can become a training environment.

Two things follow. First, work whose result can be scored unambiguously is trained preferentially and therefore becomes good first. Second, METR warns in the same breath against inferring reality from good benchmark scores: precisely because scorability is the precondition, scorable tasks overestimate performance in the real world.

For your order of projects this means: it is not complexity that decides, but the question of whether there is a yardstick. An invoice line item is right or it is not. A conversation with a supplier is neither.

And there is a second conclusion that works in your favour. The work for which there is no yardstick is exactly the work that makes your people irreplaceable. Whoever selects suppliers, resolves a conflict in the team or decides which of three offers suits the customer is working in a field where trying things out achieves nothing. That is not a morale slogan. It follows from the same curve.

What these numbers do not support

I name the limits because they are missing from every second presentation and because METR names them itself.

  • The study dates from November 2024. The models have become considerably better since then. What is robust is the shape of the curve, not the exact gap.
  • Software and research tasks were tested, not office work. The tasks were self-contained, without queries, without coordination, without company context.
  • There are orders of magnitude between fields of application. According to the METR analysis of July 2025, the time horizon, meaning the task length at which agents still solve every second task, is 40 to 100 times shorter for tasks that require looking at and operating a screen than for software and reasoning tasks.
  • In the same analysis METR concedes a possible selection bias: benchmarks tend to be created for areas in which progress can be expected in the months after their publication. Areas with slow progress therefore appear less often.

Anyone who uses such numbers to calculate for you how many jobs will disappear in your accounts department is calculating with a measurement that was made for something else. The direction is right. The transfer to your business is an analogy and not a calculation.

What the measurement does support is the direction of the curve and the reasoning behind it. That short stretches suit the agent and long ones the human follows, in my reading, from the training method and not from the accident of one model generation. That statement survives the next version jump. The factors four and two do not survive it. Take the pattern for the order of your projects, and take no number from this study into your business case.

Two axes, four fields, your order

Take your task list and plot each item on two axes. First axis: how long does a trained person need? Second axis: can someone establish in a short time whether the result is right?

  • Short and checkable: every sensible introduction starts here. Matching master data, pre-sorting documents, evaluating test records, generating texts to a fixed scheme, translations. The error shows immediately and costs little.
  • Long and checkable: worthwhile, but only after breaking it down. A process of four hours becomes six sub-steps, each with its own checkpoint. The curve above says why: over the short distance the agent is strong.
  • An example of such a checkpoint: when breaking down a quotation process, the step “pull the bill of materials from the drawing” is checkable, because the number of items can be counted against the drawing. The step “set the risk surcharge” is not.
  • Short and not checkable: the most dangerous corner. The result is there quickly, looks clean, and nobody can hold it against anything. This is where the impression of productivity without evidence arises.
  • Long and not checkable: stays with people. Quotation strategy, supplier selection, the conversation with the customer who wants to leave.

Most businesses start in the third corner, because that is where it looks most impressive. That is why the balance after six months turns out so thin.

A practical case from a manufacturing company, anonymised. There, sales had quotation texts generated for six months. Fast, well worded, everyone was satisfied. When asked whether the quotations were accepted more often as a result, there was no figure, because none had been collected beforehand. In parallel, an inconspicuous matching of master data against supplier lists was running in work preparation. There the error rate was known before and measurable afterwards. Only the second case could be evidenced to the management.

The question is not what is difficult in your business. The question is how you recognise within one minute whether the result is right. If you find no answer to that, the task is not yet ready.

Checkable does not mean checked

This is where things regularly tip over. A task is checkable, so it is automated, and then nobody checks anyway. The yardstick exists in theory and is never applied.

ISO/IEC 42001 has a control of its own for this. A.6.2.4 governs the verification and validation of AI systems and is the place where the acceptable performance limits are laid down. I ask two questions from it in every audit:

  • Which release criteria did you define in advance, and who approved them?
  • Which acceptable error rate applies to this system, and what do you derive it from?

The typical nonconformity looks like this: test results exist, release criteria and error rates defined in advance are missing. The result is judged sufficient after the fact. The control is therefore ineffective, and improvement under clause 10.2 cannot detect nonconformities in the system at all. Whoever sets the yardstick after the result measures nothing.

I know two answers to the question about the error rate well. The weak one is: we look at the results regularly and intervene if something stands out. The robust one is: two per cent of wrongly assigned items per month are permitted, collected from a sample of fifty transactions, measured since March, most recently at 1.4 per cent; if the limit is exceeded, assignment goes back to the clerks until the cause is established. The first answer is an attitude. The second is evidence.

In practice this means: the error rate is written down before the first production run, together with the question of what happens if it is breached. Two sentences are enough. They only have to be there beforehand.

What you can do on Monday

Take one area, not the whole business. Three steps, one hour.

  • List ten recurring tasks and estimate the time of a trained person for each. Minutes are enough, no time study.
  • Write behind each task how you recognise that the result is right. If you find nothing, draw a dash. The dashes are the answer.
  • Do not throw away the lines with a dash. They are the list of the work where a person has to decide, and therefore also the list on which your workforce planning rests.
  • From the lines without a dash, take the shortest. That is your first case, regardless of how unspectacular it looks.

Before you start you need two more numbers: how often the task goes wrong today, and how often it may go wrong in future. You collect the first over two weeks. You decide the second before the system runs.

Expect not to have the first number. In most businesses it is unknown how often a transaction goes wrong today, because errors are repaired silently. Collecting it feels like bureaucracy and is the part that makes the difference later. Without a baseline, all you can do after a year is claim that things have improved.

This is less exciting than an agent that writes quotations on its own. It is the part that is still running after six months and can be evidenced. And it is the part with which you will not flounder in the next audit when someone asks for the evidence.

Share: LinkedIn E-Mail

Frequently asked questions

Does this mean complex tasks are unsuitable for AI?+

No. Complexity alone does not rule a task out. Two other things matter: how long the task takes in one stretch and whether the result can be held against a yardstick. A complex but clearly checkable task is a good candidate. A simple task without any checkpoint is a bad one, because you cannot evidence the benefit later.

Can I automate long processes anyway?+

Yes, after breaking them down. The METR measurement shows that agents are strong in short stretches and fall behind with long time budgets. If you cut a long process into sub-steps, each with its own checkpoint, you are working over the short distance again. That costs preparatory work and is the difference between a pilot project that runs and one that peters out.

Are these numbers any use for a business case?+

Not directly. The study dates from November 2024 and looks at software and research tasks that are self-contained. In an analysis of July 2025, METR itself points out that fields of application differ by orders of magnitude and that automatically scorable tasks tend to overestimate real-world performance. The pattern is robust, not the factor. Your own calculation needs your own measured values.

What exactly does ISO/IEC 42001 require before a system goes live?+

Control A.6.2.4 requires documented verification and validation measures together with criteria for their use. The guidance in Annex B.6.2.4 names testing methods, representative test data, release criteria, acceptable error rates and the handling of factors that reduce performance. The timing is decisive: the criteria must be fixed before release. If they are adjusted to the result afterwards, the control is not effective.

Author & expert review: Lars Zimmermann · ISO/IEC 42001 Senior Lead Auditor & Senior Lead Implementer · ISO/IEC 27001 Lead Auditor & Lead Implementer (PECB)

Last updated: 10 November 2026. Researched and reviewed to the best of our knowledge; not a substitute for individual legal advice.

Questions about your own case?

In a free 15-minute intro call we assess where you stand on ISO 42001, ISO 27001 and the EU AI Act, honestly and without a sales pitch.

Continue reading