When AI starts building AI: keeping feedback loops under control

Since generative AI arrived in Australian organisations, the advice has been consistent: check its output. But what if the source you’re checking against is itself unchecked AI output, and nobody can trace where it came from?

Picture an organisation that rolls out an AI tool to summarise customer support calls. The pilot goes well and everyone’s happy. Then the summaries start travelling, beyond the teams with the context to check them. They’re saved into the CRM as account notes. A second tool reads those notes to draft follow-up emails. A third uses them to flag customers at risk of leaving. One summary says a customer cancelled their contract, when they’d only asked about the terms. Nobody spots it, because the note looks like every other note. The customer receives a retention offer they never asked for, and the churn model learns from the error. When a manager later asks why this customer is being treated as lost, nobody can say where the claim came from.

No single tool failed. The AI did what the “AI can make mistakes” disclaimer says it can do. The problem was that its output moved from system to system with no label, no checkpoint and no owner, and became the evidence for every decision after it.

That’s the risk now emerging as AI is used to build, test and improve other AI.

Most Australian businesses aren’t building frontier models. But the underlying question is already relevant to organisations using AI: what happens when AI-generated outputs become inputs to later decisions, systems or models?

The answer is not to avoid AI or all feedback loops. It is to avoid uncontrolled recursion: AI outputs feeding into more AI without clear provenance, validation, human accountability or a way to stop the chain.

A successful pilot may not reveal the loop

It can show that an assistant summarises documents well or that a model classifies requests accurately. It can’t show what happens when that output is reused by another system, carried into a decision or treated as authoritative by a person who can’t see how it was produced.

Testing is more complex than checking that each use case works. AI drives a new type of testing requirement: context testing. Traditional testing asks whether a system gives the right output for a given input. Context testing asks whether it still does when the context around it changes. The same model can pass a pilot and fail in production with no change to its own logic, because the data, prompts and upstream outputs it’s handed are different. AI is also probabilistic, so the same input can produce different outputs, and different models will often produce different outputs again. That’s why it’s so important to use business data points wherever possible to ground responses in the right context.

The risk grows as AI connects to more parts of a workflow:

• An AI-generated answer becomes source material for a later AI task
• A model’s recommendation triggers an automated action
• Generated or synthetic data is mixed with verified business data
• AI output is stored without a label showing its source or confidence
• A vendor update changes the behaviour of one link in the chain

This can create a feedback loop in which errors, omissions or bias are repeated and amplified. It can also make it harder to tell whether a result came from original evidence, human judgement or earlier AI output. None of this is inevitable, but all of it needs assessing.

That’s why the design of the workflow and its controls matters beyond a working pilot. Before a pilot is called successful, it should test where its outputs go next and what context they arrive in

Map where AI outputs travel

Before scaling a use case, trace the path from input to outcome. Which data enters the system? What does the AI produce? Who reviews it? Does it move into another tool, dataset or decision? Can an action follow automatically? This is a practical way to expose hidden recursion. It also shows where human review, source labelling or a technical boundary should sit.

Data protection and classification remain essential. So does provenance, people need to know whether information is original, AI-generated, AI-transformed or independently verified. Without that context, an organisation can inadvertently treat generated content as grounded truth.

For each connection in the workflow, ask:

• Is AI output allowed to become an input elsewhere?
• Is the output clearly identified and traceable to its source?
• What validation is required before it is reused?
• Does a person need to approve the next decision or action?
• How will the organisation detect changes in quality or behaviour?

The more consequential the next step, the stronger the validation should be. A low-risk draft for human editing is different from a recommendation that updates a customer record or initiates a transaction.

Make ownership clear across the chain

Recursive workflows rarely sit within one team. A business unit may own the process, data teams may manage the source information, IT may connect the systems and cyber or risk teams may set the controls. But if no one owns the whole chain, each component can pass its local check while the combined workflow remains poorly understood.

A workable operating model makes clear who can approve an AI use case, who reviews its outputs, who monitors performance and who can pause or change it. It should also define which kinds of AI-to-AI handoff are allowed, what evidence is required and when an independent evaluation is needed. That doesn’t mean every decision must be centralised. It means decision rights and escalation paths are visible before a workflow becomes business-critical. Governance needs to keep pace as the model, its connections and its level of autonomy change.

Four safeguards against uncontrolled recursion

Here are some practical steps to take to reduce the risk of uncontrolled recursion:

1. Label and track AI outputs. Preserve provenance so users and downstream systems can distinguish generated content from verified source material.
2. Set rules for reuse. Define when AI output may be fed into another AI system, dataset or decision — and when it needs human validation first.
3. Add checkpoints where consequences rise. Require review before sensitive data is used, important records are changed or consequential actions are taken.
4. Test the workflow end to end. Assess connected systems and repeated handoffs, not only the first model or prompt; monitor changes and provide a clear stop mechanism.

The objective is not to remove useful automation. It is to ensure that each step has an accountable owner, an appropriate level of evidence and a way to intervene when the output is wrong or the system changes.

From AI adoption to AI assurance

The conversation about AI return often begins with productivity, service or cost. As AI systems become more connected, another question belongs in the business case: can the organisation see and control how AI outputs shape what happens next? That is where AI assurance becomes practical. It connects governance, data readiness, security, workflow design and ongoing monitoring so organisations can understand not just what an AI tool produces, but how that output travels and influences decisions.

Interactive brings together Data & AI, Cyber, Cloud and Managed Services to help customers assess AI foundations, map risks across workflows, define controls and plan for safe scaling. The aim is not to stop AI contributing to innovation or business processes. It is to prevent unverified feedback loops from becoming part of how your organisation operates.

AI can help build the next tool, inform the next decision or improve the next process. But when its outputs feed back into the system, human accountability must not drop out of the loop.

If your organisation is connecting AI across workflows and wants to understand where those feedback loops lead, Interactive can help you assess the risks and put practical assurance around them.

Find out more about our Data and AI solutions.

Keeping feedback loops under control is one of several AI risk management issues we’re seeing right now. For more on data and AI, stay tuned to our regular insights.