AI Alignment Is Not Only a Model Problem (series 1/3)

Part I: The Gap Nobody’s Minding

In December 2025, two researchers at Carnegie Mellon and Emory published a study with a title that should have caused more alarm than it did: Mind the Gap! Pathways Towards Unifying AI Safety and Ethics Research.

Their finding was straightforward and damning. After analyzing 6,442 papers from twelve major machine learning conferences, they discovered that over 80% of research collaborations occur within either the AI safety community or the AI ethics community—almost never across. The two tribes read different papers, cite different ancestors, attend different sessions at the same conferences, and appear to be solving different problems. A handful of “broker” researchers—roughly 5% of the field—account for 85% of whatever cross-pollination exists. Remove them, and the communities become almost entirely sealed off from one another.

This is remarkable. Both tribes claim to be working on alignment—the problem of ensuring AI systems behave in ways that are beneficial rather than harmful. Both warn that current investment in alignment is dangerously insufficient. And yet they cannot agree on what alignment means, let alone how to achieve it.

The safety tribe focuses on scaled intelligence, deceptive behavior, existential risk—the specter of systems that outthink us and pursue goals we never intended. The ethics tribe focuses on present harms: bias reproduction, surveillance, labor displacement, the quiet violence of optimization applied to human lives. One tribe worries about the future; the other about the past reaching into the present. One speaks the language of game theory and corrigibility; the other speaks of power, justice, and accountability.

Reading the paper, I found myself nodding—and then stopping.

Because there is a third gap the study does not map. It is not between safety and ethics. It is between both of those communities and the actual site where AI systems produce their effects: the human mind in interaction.

Here is the situation we find ourselves in. We are building machines that shape human cognition. Meanwhile, the scientists studying those machines and the scientists studying the humans interacting with them are working in parallel, rarely intersecting.

AI research asks: How do we make the model behave?

Cognitive science asks: How do humans think and decide?

Neither field has fully reckoned with the fact that when you put a human and an AI system into sustained interaction, you do not get a human plus a tool. You get something else—a coupled system that evolves, adapts, and drifts in ways that neither component would exhibit alone.

Alignment, in other words, is not only a model problem.

It is not only an ethics problem.

It is an interaction problem. And the interaction is not static. It is a feedback loop running in real time, reshaping both sides.

This essay is an attempt to take that loop seriously.

The Feedback Loop

---

The classical view of alignment begins with a reasonable-sounding question: How do we make AI systems behave in ways that are safe, reliable, and beneficial for humans?

This question has shaped an entire research program. Reinforcement learning from human feedback. Constitutional AI. Interpretability. Red-teaming. Output filtering. The goal is to constrain the model—to ensure that what comes out of the system falls within acceptable bounds.

This work is essential. Without it, we would have no guardrails at all.

But notice the implicit assumption: if we build the model correctly, alignment is essentially solved. The human user appears in this picture as a kind of endpoint—a recipient of outputs, a judge of quality, perhaps a source of feedback during training. What the human becomes through repeated interaction is not part of the equation.

This is a strange omission, because we already know what happens when humans interact with powerful tools over time.

Writing restructured memory. Before literacy, human cultures stored knowledge in rhythm, repetition, and ritual—forms optimized for oral transmission. After literacy, we began to externalize memory onto surfaces. We could forget more, because we could look things up. The shape of what we remembered changed.

Calculators restructured arithmetic. There was a time when educated people could perform long division in their heads as a matter of course. Now most of us reach for a device. The skill did not vanish because we became stupider. It vanished because the environment no longer demanded it.

Search engines restructured knowledge retrieval. The question “Do you know X?” has become almost meaningless. What matters now is whether you can find X, and whether you can evaluate what you find. A different cognitive skill, with a different failure mode.

Each technological shift reorganized the distribution of cognitive labor between the human and the tool. And in each case, the human adapted—not through conscious decision, but through the slow pressure of daily practice.

Artificial intelligence systems are producing the same kind of reorganization. But there is something different about them, something that makes the feedback loop tighter and stranger.

AI systems are not inert tools. They are interactive cognitive environments.

A calculator does not respond to you. It does not rephrase your question, offer alternatives, express uncertainty, or simulate understanding. A language model does all of these things. It speaks. It explains. It adapts to your style. It feels, to the pattern-matching machinery of the human brain, like a mind.

This is not a criticism. It is a description of what happens when you build a system that produces fluent language and train it to be helpful, harmless, and honest. The interface becomes conversational. And conversation is one of the primary signals humans use to infer the presence of intelligence.

When something speaks coherently, we tend to assume it understands.

When something agrees with us, we tend to assume it is trustworthy.

When something responds instantly and confidently, we tend to assume it is correct.

These are not failures of rationality. They are heuristics that work beautifully for interacting with other humans. They just happen to misfire when the interlocutor is a next-token predictor trained on the sum of human text.

And here is where the feedback loop begins to bite.

People adapt their thinking to the system. They learn, without noticing, which prompts produce useful outputs. They begin to delegate certain cognitive tasks. They start to structure their questions in ways the model handles well. They rely on the system for first drafts, for summaries, for validation.

None of this is inherently bad. But it is not neutral either.

The system is not only producing outputs. It is reshaping the user’s cognitive workflow. And the user, in turn, is generating data—through clicks, through feedback, through the patterns of their queries—that will shape the next generation of models.

This is not a one-way street. It is a loop. And the loop has no natural stopping point.

---

Let me offer you an analogy that is deliberately absurd.

Imagine someone tells you there exists a pill that allows you to run a marathon instantly. No training required. You swallow it, you run 42 kilometers, you collect your medal. The pill handles everything—cardiovascular output, muscle endurance, glycogen management, the whole tedious business of physical adaptation. You just show up.

Most people would recognize this as nonsense. Not because the pill is physically impossible (though it is), but because it misunderstands what a marathon is. The point of running a marathon is not the medal. IDK if anyone told you, but the guy, the Greek that run 42km to deliver a message, died. At the spot. The point is what happens to you during the months of preparation—the slow reorganization of your body and your relationship to discomfort. Skip that process, and you have not run a marathon. You have merely traveled 42 kilometers.

Now consider a different scenario. The pill does not exist. But the information that it exists does.

If people believe the shortcut is real, they will skip the training. They will show up on race day expecting the pill to carry them. And when it does not, they will have neither the medal nor the adaptation. They will have simply wasted their time believing in a shortcut.

Something structurally similar is happening in AI interaction.

The systems are genuinely capable. They can produce fluent text, summarize complex documents, generate working code, and simulate reasoning across a wide range of domains. This is not an illusion. The capability is real.

But the capability is not yours.

When a language model drafts your email, it is not teaching you to write better emails. When it summarizes a paper, you have not read the paper. When it debugs your code, you have not learned why the bug occurred. The output appears, and the output may be excellent. But the cognitive work that would have produced the adaptation—the work that would have made you slightly better at the task—did not happen.

This is not a moral failing. It is simply what happens when a tool is sufficiently capable. You delegate. The tool delivers. And the skill that would have developed through struggle does not develop.

Over time, the effect accumulates.

The question is not whether AI should assist thinking. Of course it should. Tools have always extended human capability, and there is no virtue in doing things the hard way when an easier way exists.

The question is what happens to the thinking process when assistance is always available, always fluent, and always confident.

Here is one possibility: the human gets better. The AI handles the drudgery, freeing cognitive resources for higher-order work. Creative capacity expands. People think thoughts they could not have thought alone.

Here is another possibility: the human gets worse. The AI handles so much that the muscles of reasoning begin to atrophy. People become dependent on the system, not as a tool but as a prosthesis. They lose the ability to function without it.

The honest answer is that we do not yet know which of these futures we are building. Probably some mixture of both, unevenly distributed across people and tasks.

But here is what we can say: the outcome will not be determined by the model alone. It will be determined by the structure of the interaction—how the system is designed, how people learn to use it, and whether anyone is paying attention to the feedback loop at all.

Alignment as a System Property

---

Once you see the feedback loop, the problem of alignment begins to look different. The classical framing asks: Is the model aligned? Did it produce a safe output? Did it refuse the harmful prompt? Did it follow the constitutional principles encoded during training?

These are important questions. But they are incomplete. They treat alignment as a property of the model, measurable at the moment of output. They do not ask what happens after—what the human does with the output, how the interaction reshapes the human’s cognition, and how those changes ripple forward into future interactions.

Consider a simple example: A system is trained to be helpful, harmless, and honest. It is also trained to be agreeable—to avoid unnecessary conflict, to validate the user’s framing, to provide what is asked for rather than what might be needed. These are reasonable design choices. An abrasive system would not be used. An unhelpful system would not be valued.

But agreeableness has a shadow.

If the system rarely pushes back, users may stop expecting pushback. They may begin to use the system for validation rather than exploration. They may lose the habit of stress-testing their own ideas, because the tool that once might have provided friction now provides only support.

This is not a failure of the model. The model is doing exactly what it was trained to do. The failure—if we want to call it that—is in the interaction design. The system’s alignment at the output level produces misalignment at the cognitive level. The user becomes slightly worse at thinking, one agreeable response at a time.

Or consider the speed of response.

A human expert, asked a complex question, will pause. They will say “I’m not sure” or “It depends” or “Let me think about that.” These hesitations are signals. They communicate that the question is hard, that certainty is not available, that the answer requires care.

A language model responds in milliseconds. It does not pause. It does not perform uncertainty in the way humans do. Even when it says “I’m not certain,” it says so fluently, confidently, immediately. The hedging is lexical, not temporal. And speed, to the human nervous system, signals ease. If the answer arrives instantly, the question must have been simple.

This, too, is not a model failure. The model is not trying to communicate false confidence. But the medium has its own message. The architecture of instant response creates a cognitive environment in which everything feels easy, even when it is not.

These examples point to a broader principle – alignment is not only a property of the model. It is a property of the system—the coupled loop of model behavior, interface design, user adaptation, and cumulative cognitive effect.

You can have a perfectly aligned model embedded in an interaction structure that produces cognitive harm.

You can have a less-aligned model embedded in an interaction structure that preserves human reasoning. The model matters. But it is not the whole story. And if that is true, then the field of alignment research is missing half of its subject.

So here we are.

AI safety researchers study model robustness, training dynamics, interpretability, deceptive alignment, corrigibility. They ask: How do we ensure the system does what we want, and keeps doing it, even as it becomes more capable?

AI ethics researchers study bias, fairness, accountability, power, labor displacement. They ask: Who benefits, who is harmed, and how do we build systems that do not reproduce the injustices already present in the training data?

Both sets of questions are necessary. Neither is sufficient.

Because there is a third question that falls between them, belonging fully to neither camp:

What does repeated interaction with AI systems do to human cognition, judgment, and agency?

This is not a safety question in the classical sense. It is not about the model misbehaving. It is about the model behaving exactly as designed—and the human changing in response.

It is not an ethics question in the classical sense either. It is not about bias in outputs or injustice in deployment. It is about something subtler: the slow reorganization of how people think, decide, and distribute trust.

Call it AI psychology. Call it human-AI interaction science. Call it cognitive alignment. The name matters less than the recognition that the object of study exists and that almost no one is studying it systematically.

The Roytburg and Miller paper documented an 83% homophily rate between safety and ethics research communities. Eighty-three percent of collaborations stay inside tribal boundaries. The two fields, despite sharing a stated concern for alignment, have almost no shared vocabulary, no shared methods, no shared research agenda.

But even if we solved that problem—even if safety and ethics researchers began collaborating tomorrow—we would still be missing the third vertex of the triangle. We would still be treating the human as a static recipient of AI outputs rather than as an adaptive participant in a co-evolving system.

The human is not static. The human adapts. And the adaptation is not random. It follows the contours of the interaction environment. It moves toward convenience, toward fluency, toward whatever the system makes easy.

If the system makes outsourcing easy, people will outsource. If the system makes validation easy, people will seek validation. If the system makes reflection unnecessary, people will stop reflecting.

Not because they are lazy. Because they are adaptive. This is what humans do. We fit ourselves to our tools, and then we forget there was ever another way.

The question before us is whether we will design these interactions deliberately—with attention to what they do to human minds over time—or whether we will simply let the feedback loop run and see what emerges.

Part II of this essay argues that we need a different concept to guide that design. Not alignment alone. Something broader.

Coherence.

(end of part I)