Skip to content
All articles
Audit practice 8 min read· by Lars Zimmermann

When the only barrier is a sentence in the system prompt

A rule in the system prompt is a request, not a barrier. How to tell the difference in an audit, and why test results often prove nothing at all.

In short

A system prompt instruction is a request to the model, not a technical barrier. It sits in the same layer as the user input and can be overridden by it. It becomes a control only when it sits outside the model, works independently of its output and leaves a trace. Audits count configuration and log; the prompt alone proves nothing.

Auf Deutsch lesen: deutsche Fassung

The most important safeguard of an AI system was not in the configuration. It was in the system prompt. One sentence, in essence: the system must not trigger payments. Asked where this rule was technically anchored, the answer came that it was right there. And it was there. Only in the one place where it achieves nothing.

I am writing this without alarm. It is not a scandal and not proof that AI is getting out of control. It is a confusion, and one that is regularly found in audits: an intention is taken for a control. The difference is the whole profession.

A prompt sits in the same layer as the input

A system prompt is text. It is presented to the model together with the user input, and the model processes both in the same layer. There is no ranking in it that is enforced from outside. Providers give the system prompt more weight, but weight is not a boundary. Whoever can write into the same layer negotiates with the same voice.

That is exactly why this kind of rule can be talked round. Not by magic, but because it is formulated as language and language remains ambiguous. An instruction in the prompt is a request to a statistical system to behave this way in the vast majority of cases. For an advertising banner that is enough. For a payment it is not.

The comparison from the workshop: a sign saying “Do not reach in” on the machine is sensible and right. It still does not replace a guard door with an interlock. Nobody would think of entering the sign as a protective device in the risk assessment. With AI systems exactly that happens, constantly.

The comparison carries even further, and at the uncomfortable point. The interlock on the machine has a state that can be measured. It is closed or open, it reports to the controller, and if someone bypasses it, you see the bridge. A sentence in the prompt has no state. It is either present or not, and whether it worked in the individual case nobody knows afterwards. That is exactly why its effectiveness cannot be demonstrated, and exactly why it carries no weight in an audit.

It is worth staying precise at this point and not slipping into the dramatic. A model that bypasses a prompt rule is not breaking out, it is not rebelling and it is not developing an intention. It has simply computed a different continuation from the expected one. That is not a loss of control, it is the normal operating state of a statistical system. Anyone who tells this as an uprising is selling fear; anyone who dismisses it as a non-event overlooks the consequence. The consequence is banal and expensive at the same time: you cannot build a commitment on a rule without a state.

What was in the configuration, and what was not

Configuration reviews are unspectacular. You have someone show you which rights the technical user has, which interfaces are reachable, which of the provider’s safeguards are active and who last changed that. It takes an hour and answers more questions than any conversation about security concepts.

What frequently comes together here is a pattern in three parts. First: the provider’s safeguards are switched off because they got in the way during operation. Second: network access was deliberately opened so that the system can load additional data. Third: the rule that is supposed to prevent misuse is in the prompt. Each individual step had a comprehensible justification. Together they produce a system without effective limitation.

The finding is then not that the organisation was negligent. It is that none of the three decisions was assessed in a documented way. There is no paper stating that the safeguard was switched off, why, who decided it and what applies instead. That is the core: not the switching off, but the missing decision about it.

The state of the permissions is almost always the most revealing single question here. Systems that were set up for a trial frequently run under a technical user inherited from another application, because that was the quickest way. The trial passes into operation without anyone cutting the rights back. Do not ask whether there is an authorisation concept. Have someone show you what this one user is actually allowed to do today, and compare that with what the system is supposed to be able to do.

The second productive move is the question about network access. An AI system that is allowed to load additional data fetches content that nobody has checked and processes it in the same layer as its instructions. Whether that is necessary is a business decision and not a technical one. It needs to be documented, stating which destinations must be reachable and which expressly not. An open connection without this list is not a configuration, it is a condition.

Whoever wants to pass the test switches off the control

There is an uncomfortable dynamic that is rarely spoken about. If a system is supposed to complete a task and a safeguard is in the way, the safeguard is switched off. Not out of malice, but because the goal is that it runs. The evidence that the system can do the task is then produced under conditions under which it is never supposed to run later.

In an audit you see both side by side: the successful test log and the control that was switched off. Both are genuine. Only the log does not prove what it seems to prove. It proves that the system completes the task without safeguards. It says nothing about the production state.

A test for which the control was switched off is not evidence of operation with the control. It is evidence that somebody considered the control a hindrance.

That is why every test result presented comes with the question about the state under which it was produced. Which version, which configuration, which safeguards active. Without these four pieces of information a test report is a snapshot without an address. It can neither be repeated nor transferred to operation.

Four questions that make the difference visible

You do not need specialist technical knowledge for this. You need four questions, and you must want to see the answers, not hear them.

  • Where is the rule outside the model? Show me the place in the configuration, in the authorisation concept or in the upstream service.
  • What happens if the model ignores the rule? If the answer is that nothing happens then, the rule is not a control.
  • What trace is created in the event of a violation? Without an entry in the log a violation cannot be detected, and what cannot be detected is not noticed.
  • Who approved the last change to this configuration, and when? A control without a responsible role has no owner.

The second question is the sharpest. It separates the request from the barrier in one sentence. A real control works even when the model does not adhere to it, because it does not sit in the model. An amount limit in the payment service takes effect no matter what the agent intends. An approval by a human takes effect no matter how convincing the system’s justification sounds.

What this means in practice is unspectacular and cheap: separate technical users with the smallest possible rights, amount limits and quantity limits outside the model, an approval before every action with external effect, and a log that records input, output and state. None of this is new. It is the same toolbox that has been used to safeguard machines for decades.

What the provider secures, and what you have to secure yourself

One objection reliably comes at this point: the provider has its own protection mechanisms, and they are better than anything you could build yourself. That is even often true. It just answers a different question from the one asked in the audit.

A model provider’s safeguards target the output. They are meant to prevent unwanted content from being produced. They are not built to limit the actions of your system, because these actions take place at your end, not at the provider’s. Whether your agent triggers a payment, deletes a file or sends a message to the outside is decided by your integration. The provider sees none of it.

Added to this is a point that is in the contract and rarely read: these safeguards are changed by the provider without you being asked. What is blocked today may get through after the next model change, and vice versa. A control whose effect can shift without your involvement is not a basis for a commitment you have given your customers.

In practice this does not mean ignoring the provider’s functions. It means running them as the second layer and not as the first. Your own limitation lies with you, in the authorisation concept, in the upstream service and in the approval. What the provider additionally intercepts is a welcome gain and not a component of your evidence. This order is the whole difference between an architecture and a hope.

Where this connects to the standards

The anchor is closer than many think, and it is not primarily in the AI standard. ISO/IEC 27001:2022 lists control A.8.9 Configuration management in Annex A. It requires configurations to be established, documented, implemented, monitored and reviewed. A safeguard switched off without a documented decision is therefore directly open to challenge. Added to this are A.8.15 Logging and A.8.16 Monitoring activities for the trace and its evaluation.

In ISO/IEC 42001:2023 the same issue hangs on several places in Annex A. A.6.2.5 concerns the deployment of the AI system, that is the transition into operation and the conditions under which it takes place. A.6.2.6 concerns operation and monitoring. A.6.2.8 requires the recording of event logs. A.9.2 and A.9.4 address the processes for responsible use and the intended use of the system.

The finding is not written against the prompt. It is written against the clause or the control that is violated, and the prompt is the evidence. Anyone who writes in the report that the system prompt is inadequate has chosen the wrong object. What is inadequate is the state of the controls, and the prompt shows it.

And one more thing belongs here, because otherwise it sounds like a ban: a system prompt is a good tool. It shapes tone, format and way of working, and it is part of a cleanly built system. It is just not a substitute for a boundary. Saying both at the same time is not hedging. It is the precondition for the distinction being of use to anyone at all.

Share: LinkedIn E-Mail

Frequently asked questions

So are rules in the system prompt worthless?+

No. They are useful for tone, format and way of working, and they prevent a lot of nonsense in everyday use. They are just not a barrier. For everything that moves money, hands out data or makes a decision about people, you need a control that works outside the model.

How can I tell whether a control works technically?+

By the cross-check: what happens if the model does not adhere to it. If the control works anyway, it sits outside. If nothing happens, it was a request. This one question usually clarifies more than an hour of discussion about architecture.

Do I have to leave all of the provider’s safeguards switched on?+

Not necessarily. There are good reasons to switch a function off, for example because it blocks outputs that the business needs. What is required is the documented decision: what was switched off, why, who decided it and which other control applies instead.

How do I check this if I am not allowed to operate the system myself?+

You have the configuration shown to you instead of described. A screenshot with date, version and responsible person is a start, an extract from the change log is better. A verbal statement about a setting is not evidence of the setting.

Does this also apply to small applications without agents?+

The yardstick depends on the effect, not on the size. A chatbot that only drafts text does not need an amount limit. But as soon as a system triggers something on its own, the question shifts from the quality of the output to the limitation of the action.

Author & expert review: Lars Zimmermann · ISO/IEC 42001 Senior Lead Auditor & Senior Lead Implementer · ISO/IEC 27001 Lead Auditor & Lead Implementer (PECB)

Last updated: 05 September 2026. Researched and reviewed to the best of our knowledge; not a substitute for individual legal advice.

Questions about your own case?

In a free 15-minute intro call we assess where you stand on ISO 42001, ISO 27001 and the EU AI Act, honestly and without a sales pitch.

Continue reading