Proof Over Claim: Fact-Checking Viral AI Statistics

I keep running into AI numbers that sound too good to be true, and most of the time they are. Catchy, precise, with a decimal place, often presented as fact in a podcast or a talk. And almost always the one thing that matters is missing: a verifiable source. This is my assessment, not legal advice, but as an auditor I cannot put it any other way. A number without a source, a definition, and a scope is not evidence. It is a feeling with a decimal place. I have brought along four widely cited AI figures and checked them. Not one of them delivers what it promises.

Why catchy numbers work so well

Precision reads like truth. A vague statement such as many companies use AI convinces no one. Exactly 8 percent higher quality or seven times as much sounds like a measurement, like a study, like substance. That is precisely why such numbers get used so eagerly, especially where something is being sold: a training course, a tool, a consulting service. The number creates pressure to act: if others are already this far ahead, I have to move now. The trick works because almost no one asks. So let us ask.

Example 1: Tipping the AI yields 8 percent higher quality

This number is often attributed to a large study by Meta. Both parts are wrong. The famous tipping trick, promising the AI money so it tries harder, traces back to a viral experiment on X by a user going by the name thebes in late 2023. It was not a scientific setup, and what was measured was mainly the length of the answer, not its quality. It had nothing to do with Meta.

The real 8 percent figure comes from an entirely different piece of work: EmotionPrompt, by researchers at Microsoft and the Chinese Academy of Sciences. That study was not about tipping but about emotional phrases such as this is important for my career. And even that effect is small, unstable, and has largely disappeared in newer, more heavily trained models. A narrow, uncertain lab finding turns into a hard sales argument on stage. That is how two different things become a false third claim.

Example 2: Seven times as much, according to Microsoft

Sometimes a company name alone is treated as evidence. Microsoft supposedly found that something is seven times faster or seven times as much. I checked the obvious source, the Microsoft Work Trend Index, across the 2024 to 2026 editions. This specific sevenfold figure is not in it. That does not mean nobody, somewhere, ever said something similar. It means that anyone selling it as a Microsoft fact should be able to name the exact reference. If they cannot, it is not evidence, it is a borrowed name.

Example 3: AI writes more than 50 percent of Microsoft's code

Here too, a close look pays off. What Satya Nadella actually said was more measured: in some projects, roughly 20 to 30 percent of the code came from AI, not more than half of all code. The frequently cited more than 50 percent was a projection by Mark Zuckerberg for Meta, that is, an expectation for the future, not a measured present value. Two companies, two statements, merged into a single number in the retelling.

On top of that comes a subtle but important distinction. AI-generated, initiated, reviewed, and owned by a human is not the same as written by the AI. Anyone who equates the two turns a tool into an author and, without anyone noticing, shifts the responsibility.

Example 4: The AI gets lazy in December

A persistent myth claims that a well-known model becomes measurably more sluggish in December, because it supposedly picked up a kind of winter break from its training data. The origin is a thread on X from late 2023. The effect could not be reliably reproduced in clean re-tests, and the analysis was statistically shaky. The maker never confirmed a winter cause. A good story, but not a finding.

A number without a source, a definition, and a scope is not evidence. It is a feeling with a decimal place.

The one question that decides everything

You do not need to be a statistician to protect yourself. One question is enough: where is that stated, and what exactly is being counted? Where is that stated points to the primary source. Not a screenshot, not another podcast, but the study, the annual report, the original statement. What exactly is being counted points to definition and scope. Fifty percent of what, measured how, for whom, over what period. The moment either of these two answers is missing, the number is worthless as a basis for a decision.

This is not academic luxury. Anyone who builds staffing, budget, or an entire AI strategy on a borrowed number is steering by a gauge that is not connected to anything. In an audit the stance is the same as on the test bench: the evidence first, then the statement. Proof over claim is not nitpicking. It is the difference between a decision and a gut feeling in a suit.

My clear position: anyone who presents you with exact AI numbers and, when asked for the source, answers only with I think so or I heard it somewhere, is not selling you knowledge, they are selling you urgency. Take the number, but not as a fact. Take it as a claim still waiting for its evidence.