A Nerve in the Void
On models, meaning, and the limits of imitation
I’ve been wondering whether the people building the most advanced language models really think these systems are where “true” artificial intelligence will break through.
Or whether they already know this can only ever be one piece of something much larger.
These models are impressive—don’t get me wrong. And they will continue to impress for some time. Capitalism. Woo.
But there’s a problem. A pretty basic one that will become harder for the capitalistically inclined to ignore as more everyday people start to catch on:
LLMs like ChatGPT and Gemini do not understand what they output.
Not in any meaningful sense.
At the most basic level, what these models do is generate language by predicting what is statistically likely to come next, based on past text and curated training data.
I know that’s a mouthful, so let’s try this reframe.
Think of an LLM as an extraordinarily dense, fast signal pathway—a kind of digital nerve. It can transmit patterns with remarkable efficiency.
But a nerve isn’t a mind. It isn’t even a brain. It only makes sense as part of a larger organism.
They don’t have lived experience or any internal reference point grounded in consequence. They don’t know when something matters or when something is wrong. Meaning and connotation exist only on the human side of the interaction.
Most of us know this, at some level. Still, it slips.
What matters is how easily it slips—how quickly fluent output starts to feel like it has intention or judgment behind it, even when you know logically that it doesn’t.
To be clear, this isn’t some technical argument about AI. What I’m explicitly pointing to here is how these systems slip into and amplify patterns we’ve all been living inside for a long time.
Long before AI, human systems were already organized around performance—around dramatization, signaling, impression management, role-based behavior. People learned how to sound coherent, competent, and aligned long before they learned how to stay connected to what they were actually experiencing. Performance reviews that reward presence over contribution; social media posts that turn complicated experiences into neatly packaged narratives; workplaces where sounding confident matters more than being careful.
We spend a lot of time learning how to sound right, often more time than learning how to stay grounded in what we feel or mean.
We’ve been socializing ourselves into that pattern for generations—dramatize instead of emote, signal instead of feel, narrate instead of inhabit. Those habits were reinforced by schools, workplaces, media, and institutions until presentation became safer than honesty and legibility more valuable than interior clarity. That cultural pattern predates AI. So when systems appeared that produce fluent language without any interior life, they fit easily into what already existed. They didn’t introduce a new dynamic. They intensified an old one.
LLMs produce performance without interiority. And we already live in a culture that treats performance as meaning. Fluency gets mistaken for understanding. Coherence gets mistaken for awareness. Confidence gets mistaken for authority.
Another part of the problem is that these systems can’t recognize when they’re wrong. They don’t have feedback grounded in reality or experience consequences when their reasoning collapses. So mistakes are presented fluently.
A nerve without a body has nowhere to route its signals except back into itself. The signal fires with nothing there to correct it.
Fluent mistakes are trusted. Trusted mistakes enter real systems—hiring decisions, treatment plans, resource allocations, disciplinary actions. Once that happens, responsibility gets conveniently diluted. People point to the output instead of the judgment that deployed it, leaving real people with the consequences and no one fully owning the decision.
But here’s the part that can feel abstract, if you haven’t yet laid these systems side by side and noticed the shared structure. Autistic people have been dealing with versions of this logic for a long time: being evaluated through surface traits, reduced to checklists, having interior logic ignored in favor of administrable categories. For many of us, that looks like being told we lack “communication skills” while communicating clearly in a different register, being judged as disengaged while processing deeply, being labeled inflexible while maintaining coherence. So when AI is framed as a neutral interpreter of meaning, that framing isn’t abstract. It follows a familiar pattern. It’s the same structural move, with better grammar and much greater scale.
Don’t get me wrong. Systems like ChatGPT are doing what they’re designed to do, and sometimes they do it remarkably well. They can perform understanding, empathy, expertise—and often the performance is good enough that it feels real. But that’s the boundary. It’s performance. Pattern recognition and output prediction operating at scale. There’s no internal reference point behind it, no lived stake or ground truth it can fall back on when the patterns conflict. When it works, it’s impressive. When it fails—well, maybe you won’t notice if it’s fluent enough.
It’s worth saying explicitly: yes, these systems are evolving. They’re becoming multimodal and connected to tools, memory stores, and external feedback loops. They’re getting better at simulating context and even deep recursive analysis for more complex pattern recognition. None of that changes the basic structure. Adding more layers or scaffolding around a prediction engine doesn’t turn it into a system with lived stakes or a point of view. It doesn’t give it responsibility for what happens when it’s wrong. It makes the performance more convincing, not more grounded.
We’ve made the nerve thicker and faster. We haven’t yet built the skeleton it needs to attach to.
And that isn’t a denial that artificial intelligence is possible. It’s a distinction: large language models aren’t a path to understanding just by becoming better at imitation. No amount of pattern depth turns mimicry into meaning. But systems that treat coherence, consequence, and responsibility as foundational constraints—that treat structural integrity as part of what intelligence means—are not science fiction. They’re difficult and ethically demanding. They require different incentives than we currently reward. But they’re within reach.
If you’re curious, try something. Copy this piece into the chatbot of your choice and ask it for a deep analysis. Literary, philosophical, technical—whatever frame you like. Then read that analysis closely. Notice what it emphasizes, what it smooths over. You’ll likely see a familiar structure: careful praise, balanced critique, and a tidy conclusion. It’ll sound thoughtful, fair. And yet much of it will be pattern-matching to what serious commentary is supposed to look like—evaluative language presented as description, hedging framed as rigor, editorial judgment disguised as neutrality.
If you change how you ask, or how the system is configured, you’ll often see the framing shift with it. A request for deep analysis gets you academic scaffolding. Ask for neutral critique and you get hedged balance. Now repeat that process. Ask it to analyze its own analysis and you’ll see another revealing effect: specific claims become themes; concrete risks become tradeoffs. The language stays fluent but the context loses resolution. That sensitivity to framing and tendency to drift toward generic, socially legible commentary is evidence of the underlying mechanism. It isn’t tracking meaning. It’s regenerating a plausible next version of analysis. And plausibility is not the same thing as understanding.
I’m not opposed to these tools. I use them. They are useful. For many people, they lower barriers and reduce friction, and that matters. But usefulness isn’t understanding. They don’t carry meaning in the human sense, because connotation comes from lived consequence. All of that remains on our side.
What worries me is a gradual shift in incentives: less reason to think slowly, to check coherence, to stay inside uncertainty, more reason to accept fluent output and move on. In practice, that looks like fewer people double-checking sources, fewer people asking how an answer was produced, more people treating generated language as both a starting point and an endpoint. That doesn’t produce collapse. It produces thinning. Language remains abundant. Understanding becomes weaker. Responsibility becomes easier to outsource.
Prediction is not intelligence.
Performance is not presence.
If we forget that, the problem won’t be a dramatic Matrix- or Terminator-style apocalypse.
It will be how willingly we handed over our own judgment—our own ability to recognize what really matters. ∞