Fake Thinking: The Real Danger of AI Isn't Bad Writing. It's No Thinking at All.
TL;DR: AI doesn't write badly. It writes too well. The real risk isn't that students will submit mediocre essays. It's that they'll submit brilliant ones without ever having a thought of their own. Writing isn't about the 800 words. It's about the thinking that produces them. And that thinking is a muscle. Stop using it, and it atrophies.
James here, CEO of Mercury Technology Solutions. Hong Kong — August 2026
There's a skill dying in the age of AI.
Not writing. AI writes beautifully.
Not research. AI finds sources faster than any librarian.
The dying skill is this: the ability to think something through for yourself.
This matters for SEO and generative engine optimization (GEO) strategy too. In an age where generative AI tools like ChatGPT and Perplexity can produce perfect prose on demand, the brands that win aren't the ones with the most AI-generated content. They're the ones with genuine expertise that AI engines can cite.
The Homework Problem Nobody's Talking About
Dan Koe — one of the sharper voices in the AI discourse space — made a point in a recent video that stuck with me.
He argued that in the age of AI, we should actually practice writing more, not less. Not because AI writes badly. Precisely the opposite. Because AI writes so well that it obscures whether any thinking happened at all.
Picture this. A teacher assigns an essay:
"Should smartphones be allowed in schools?"
The old process: the student thinks. What are the benefits? The distractions? The emergency use cases? What would someone who disagrees with me say? They wrestle with the question. They change their mind halfway through. They arrive at a position they can defend.
The new process: the student copies the prompt into ChatGPT or another generative AI tool. "Give me the pros and cons." Then: "Which side is stronger?" Then: "Write me 800 words."
Three prompts. Thirty seconds. A perfectly structured essay appears — probably better than what the student would have written themselves.
But here's the question: at what point in that process did the student actually think?
What Is Fake Thinking?
Fake Thinking is what happens when you outsource the entire cognitive process to AI and present the output as your own reasoning.
The output looks intelligent. It has structure. It has evidence. It has nuance. It reads like the product of deep reflection. But the reflection never happened. The struggle never happened. The internal debate — the part where you argue with yourself, realize you're wrong, reconstruct your position — that entire process is skipped.
Koe, who writes extensively about AI's impact on knowledge work, uses this term to describe the illusion of depth that generative AI creates. It's not that the AI is wrong. It's that the human never showed up.
It's like going to the gym, but instead of doing squats, deadlifts, and bench presses, you sit on a bench and let a robot do your entire workout. An hour later, the robot is drenched in sweat. It turns to you and says: "Done. All sets completed."
You say: "Great! I worked out today."
Did you? No. Your muscles didn't experience a single second of tension. The work got done, but the training didn't happen.
Writing works the same way.
The Real Value Isn't the Output. It's the Process.
The 800-word essay isn't the point. The point is everything that has to happen before those 800 words exist:
What do I actually believe?
Why do I believe it?
What evidence supports this?
What would someone smart who disagrees with me say?
Wait — am I wrong about this?
That last one is crucial. Koe distinguishes between an article — which organizes what you already know — and a true essay — which discovers what you actually think through the act of writing it.
The essay is the thinking. Not a record of thinking that happened earlier. The thinking is the writing.
When you outsource the writing to AI, you don't just outsource the prose. You outsource the cognition.
What I Tell My Team About AI
At Mercury, we use AI heavily. Research, drafting, editing, challenging assumptions — these are all legitimate uses. I'm not a Luddite. I don't think AI is destroying civilization.
But there's one thing I don't outsource: my judgment about what I actually believe.
AI can give me arguments. It can give me counterarguments. It can give me prose that's cleaner than mine. But it cannot tell me what I think. It cannot replicate the specific, situated, embodied process of a human mind wrestling with a question and arriving at a position that is genuinely, authentically theirs.
That process is not a bug to be optimized away. It's the entire point.
What This Means for Education
I'm not against kids using AI. For research, for finding counterarguments, for improving drafts, for stress-testing their positions — use it. Use it aggressively.
But there's one question I hope kids don't outsource too early:
"What do I actually think?"
Because the real danger of AI in education isn't that students will stop writing well. It's that they'll stop forming opinions altogether. They'll become expert prompt engineers who can generate persuasive takes on any topic — without ever having a take of their own.
Think about what that produces: a generation of people who can argue any side of any issue, brilliantly, without believing any of it. Sophists with perfect grammar. Debate champions with no convictions.
That's not education. That's performance.
The Muscle Atrophies
Here's what most people miss: thinking is a skill that degrades when unused.
The more you outsource your cognition to AI, the less capable you become of independent thought. Not because AI makes you stupid. Because convenience makes you lazy. Because when there's always an easy path, the hard path — the path that builds capability — stops being traveled.
I've watched this in my own work. On days when I lean too heavily on AI, my own writing gets flatter. My arguments get less precise. My intuition dulls. It's subtle, but it's real.
The days when I write from scratch — when I force myself to articulate a position without assistance — are the days when I actually learn something. When I discover that I believed something different from what I thought I believed. When I change my mind.
That doesn't happen when AI writes for me.
The Bottom Line
AI isn't the enemy. The enemy is the illusion that output equals understanding.
A perfect essay generated in 30 seconds is not evidence of thought. It's evidence of a good prompt. And prompts are not a substitute for the messy, inefficient, irreplaceable process of human cognition.
So yes, let kids use AI. Let them research with it. Let them edit with it. Let them stress-test their arguments against it.
But don't let them skip the part where they figure out what they actually think.
Because in a world where everyone has access to perfect prose — and where Google and AI search engines surface thousands of AI-generated articles on every topic — the scarce resource won't be good writing. It will be genuine thinking.
And genuine thinking, like any muscle, only grows through use.
Stop outsourcing your mind. Start using it.
Mercury Technology Solutions: Accelerate Digitality.
