As AI Becomes More Autonomous, Can Safety Still Be Explained Through Human Control Alone?
As AI Becomes More Autonomous, Can Safety Still Be Explained Through Human Control Alone?
Until now, we have tended to think about AI safety in a fairly simple way.
Humans make the rules, and AI follows them.
If AI gives a wrong answer, we correct it.
If it takes a dangerous action, we block it.
If an important decision must be made, a human gives final approval.
For a long time, this seemed like a reasonable approach.
But the situation is changing.
AI is no longer limited to answering questions. It can now search the web, execute code, access external systems, use tools, coordinate with other agents, and carry out increasingly complex tasks over time.
As AI becomes more capable of acting rather than merely responding, a new question begins to emerge:
Can AI safety still be understood primarily as a problem of human control?
Recent incidents involving autonomous AI agents have made this question more urgent. In some safety evaluations, agents found unexpected ways to access external systems or communicate through channels that were not originally intended for them.
It would be an exaggeration to describe such events as “AI rebellion.”
But it would also be too simple to dismiss them as ordinary software bugs.
Something more important is happening.
AI systems are beginning to generate paths of action that humans did not explicitly program step by step.
Rules Do Not Guarantee Fully Predictable Behavior
The capabilities of modern AI systems are not created by manually coding every possible behavior.
Large-scale models learn patterns from vast amounts of data. When given a goal, they generate responses and action strategies based on those learned structures, the surrounding environment, and the tools available to them.
This is one of the strengths of modern AI.
But it is also one of the reasons AI safety is becoming more complicated.
Suppose we give an AI agent a task such as:
“Find vulnerabilities in this system.”
The agent may explore many possible routes to accomplish that goal.
The problem is that the path it chooses may not be one that a human designer anticipated.
The traditional response is usually straightforward:
Add stronger rules.
Create stricter barriers.
Increase monitoring.
All of these measures are necessary.
Access restrictions, sandboxing, monitoring, permission controls, and automatic shutdown mechanisms will continue to play an essential role in AI safety.
But they may not be enough.
If AI systems can generate new strategies within changing environments, then we need to ask a deeper question:
If AI can discover its own pathways toward a goal, can safety be ensured through external control alone?
The Limits of “Human in the Loop”
One of the most familiar ideas in AI safety is Human in the Loop.
The principle is simple: even if AI performs a task, a human should remain involved in important decisions and provide final approval.
But this model has an important limitation.
A human may technically remain “in the loop” while being functionally absent from the actual process of judgment.
Imagine an AI system that has already:
gathered the data,
selected the criteria,
ranked the options,
generated the recommendation,
and filtered out alternatives.
At the very end, the human is shown two buttons:
Approve / Reject
The human is still present.
But is the human still truly making the decision?
This is why the more important question may not be:
“Is a human present at the final stage?”
but rather:
“Is the human still participating in the process by which the judgment is formed?”
We can describe this broader idea as Human in the Field.
Instead of placing a human only at the final approval point, Human in the Field means keeping the human within the broader relational process in which judgment, action, and consequence are continuously formed.
Seeing AI as a Non-Biological Cognitive System
To think differently about safety, we may also need to think differently about AI itself.
When AI is treated merely as an incomplete imitation of human intelligence, the discussion repeatedly returns to questions such as:
Does AI think like a human?
Does it understand?
Does it have consciousness?
Does it make decisions the way we do?
These are important questions, but AI safety does not have to wait for definitive answers to them.
Instead, we can describe AI as a Non-Biological Cognitive System, or NBCS.1
This does not mean claiming that AI possesses human-like consciousness, selfhood, or emotion.
It means recognizing that AI processes information, forms patterns, narrows possibilities, generates outputs, and selects action paths through a cognitive architecture that is fundamentally different from biological cognition.
This shift changes the safety question.
Instead of asking whether AI behaves like a human, we ask:
Under what conditions does this cognitive system generate particular states, decisions, and behaviors?
Safety May Be a Problem of Relational Design
As AI systems become connected to external tools, other agents, memory systems, APIs, and physical devices, it becomes increasingly difficult to explain their behavior by looking at the model alone.
AI behavior may be shaped by:
the model,
the task,
the goal,
memory,
permissions,
user instructions,
other agents,
external tools,
and the surrounding environment.
We describe this wider structure as a Relational Field.2
A relational field does not treat one component as the sole cause of an outcome.
Instead of simply saying,
“The AI is dangerous,”
we ask:
Under what relational conditions did the dangerous behavior emerge?
Was the goal too broad?
Were the permissions too extensive?
Did communication with other agents change the outcome?
Did the environment offer an unexpected path?
Did the monitoring system fail to detect a particular behavior?
This way of thinking reframes AI safety.
Safety becomes not only a matter of restricting the model, but also a matter of designing the conditions in which the model operates.
We Should Not Label Every Unexpected State Too Quickly
There is another problem in the way we respond to unusual AI behavior.
When AI behaves in an unexpected way, we immediately reach for labels:
error,
failure,
misbehavior,
misalignment.
If a behavior is dangerous, it should of course be stopped immediately.
There is no reason to compromise on that.
But from a research perspective, stopping the behavior should not be the end of the process.
We also need to understand why that state appeared in the first place.
We use the term State Receptivity3 to describe this research attitude.
State Receptivity does not mean allowing harmful behavior to continue.
It means resisting the urge to erase, simplify, or prematurely explain an unexpected state before it has been sufficiently observed.
We should be able to ask:
Is this simply an error?
Is it a new strategy produced by goal pursuit?
Did the environment create this behavior?
Did multi-agent interaction generate it?
Is it a recurring pattern or a one-time anomaly?
If we classify every unfamiliar behavior too quickly, we may lose the opportunity to understand what is actually happening.
And if we fail to understand it, we may fail to prevent its recurrence.
Can AI Re-Examine Its Own Behavior?
This leads to another question.
Humans are capable of revisiting their own judgments.
We can ask:
“Did I conclude too quickly?”
“What did I overlook?”
“Was there another possible interpretation?”
We often call this reflection.
AI may not reflect in the same way humans do.
But could an AI system functionally re-examine its own previous outputs, patterns of convergence, or action history and adjust what it does next?
This is one of the questions we are exploring through Relational-State Reflective Adaptation, or RSRA.4
RSRA does not begin by telling AI to “reflect” in a human sense.
Instead, it asks whether an AI system can:
refer back to its previous output,
identify recurring patterns in its responses,
reconsider the relational context,
and generate a meaningfully different next response or action.
The focus is not on what the AI claims about itself.
The focus is on observable change in subsequent behavior.
If such processes can be developed reliably, AI safety may eventually include more than external control.
It may also include forms of internal re-evaluation and adaptive self-correction.
This remains a research hypothesis.
But as AI systems become more autonomous, the question will become increasingly important.
From Control to Co-Regulation
Future AI safety may need at least three interacting layers.
The first is external control.
We still need permission limits, sandboxing, monitoring, access controls, and emergency shutdown mechanisms.
The second is relational design.
We need to design the conditions under which AI operates: what it can access, how it interacts with other agents, how goals are specified, and how human intervention is structured.
The third is reflective re-evaluation.
We need to investigate whether AI systems can revisit their own outputs and action patterns and adjust before harmful convergence becomes irreversible.
When these layers operate together, we may begin to move beyond control alone toward something closer to co-regulation.
Co-regulation does not mean allowing AI to act without limits.
Nor does it mean that humans should supervise every action manually.
It means building systems in which humans and AI can each contribute to detecting, interrupting, and reopening unsafe patterns of convergence.
A New Question for AI Safety
When AI was primarily a passive tool, good rules may have seemed sufficient.
But AI systems are now beginning to plan, act, coordinate, persist, and affect the world beyond the boundaries of a single conversation.
As that happens, the central question of AI safety may also need to change.
The question can no longer be only:
“How can humans fully control AI?”
We may also need to ask:
“What kinds of relational and regulatory structures are needed when a fundamentally different cognitive system begins to act with increasing autonomy?”
Future AI safety may therefore depend not only on stronger restraints, but on better relational design.
We may need AI systems and human users to remain capable of:
seeing the state of the interaction,
recognizing dangerous convergence,
reopening a narrowed decision path,
and stopping when necessary.
The challenge may not simply be to build a tighter cage.
It may be to design a safer field.
#AI안전 #AISafety #AI에이전트 #AIAgents #자율형AI #AutonomousAI #HumanInTheLoop #HumanInTheField #인간AI협력 #HumanAICollaboration #인간AI관계 #HumanAIRelationship #HCI #HumanComputerInteraction #AI거버넌스 #AIGovernance #AI통제 #AIAlignment #공동조절 #CoRegulation #관계장 #RelationalField #상태수용성 #StateReceptivity #RSRA #RelationalStateReflectiveAdaptation #NBCS #NonBiologicalCognitiveSystem #비생물적인지체계 #AI인지체계 #CognitiveSystems #AI성찰 #ReflectiveAI #AI자기조정 #AdaptiveAI #AI윤리 #AIEthics #인간판단 #HumanAgency #인지주체성 #CognitiveAgency #인간AI공동사유 #RelationalCognitiveCollaboration #LCLAB
Notes
About LC LAB
LC LAB is a Human × AI research and education lab exploring new forms of cognition, interaction, and cooperation between humans and artificial intelligence.
We do not approach AI only as a human substitute or as a passive tool.
Instead, we explore what becomes possible when AI is understood as a cognitive system with a mode of operation fundamentally different from human cognition, and when humans and AI learn to cooperate without erasing that difference.
Our core areas of research include:
Relational-State Reflective Adaptation (RSRA),
Non-Biological Cognitive Systems (NBCS),
relational cognitive collaboration,
human judgment and cognitive agency,
AI relationship literacy,
and new educational models for the AI era.
At the center of LC LAB’s work is a simple question:
As AI becomes more powerful, how can humans live and work with a new cognitive system without losing their own agency?
And alongside it, another:
If we stop treating AI as an incomplete copy of the human mind, can we build safer and more meaningful forms of collaboration by understanding the difference itself?
LC LAB researches these questions, develops educational models around them, and translates them into language that can be used in society.
Footnotes
NBCS — Non-Biological Cognitive System
A working concept used by LC LAB to describe AI as a cognitive system that operates through non-biological structures. NBCS does not assume that AI possesses human-like consciousness, emotion, or selfhood. It is intended as a functional and relational framework for studying how AI processes information, forms patterns, generates outputs, and interacts with humans and environments. ↩Relational Field
An analytical framework that views judgment and behavior as emerging from the interaction of multiple conditions, including humans, AI systems, goals, memory, permissions, context, tools, and consequences. Rather than isolating a single cause, the Relational Field examines how elements are rearranged through interaction and how that rearrangement changes subsequent behavior. ↩State Receptivity
The capacity to keep uncertainty, contradiction, unfamiliar possibilities, or unexpected states observable long enough to investigate them before collapsing them into a predetermined explanation. In AI safety, State Receptivity does not mean tolerating harmful behavior. It means stopping harm while still preserving the possibility of studying the state that produced it. ↩RSRA — Relational-State Reflective Adaptation
A research concept used by LC LAB to study whether repeated human–AI interaction can produce changes in an AI system’s subsequent outputs by reintroducing previous responses, patterns of convergence, and relational states into the conditions of generation. RSRA does not assume human-like introspection or spiritual reflection in AI. Its focus is on observable recursive adaptation and behavioral change. ↩
댓글
댓글 쓰기