Closed systems of thinking

No matter the environment, eventually people will adapt to it. We’ve adapted to the normalcy of carrying calculators in our pockets, though our elementary school teachers assured us this would never happen. It turns out the calculator can also do other things. For instance, it can serve as a gateway to the sum total of all human knowledge or even make phone calls.

We’ve adapted to these calculators so readily, and come to rely on them so absolutely that cultural discourse, private conversations, and perhaps even cognition itself are now mediated by them. Most trains of thought take a detour through the device before arriving at their destination, whether that be to retrieve information from a web search, from a photo, from a tweet, from ChatGPT, etc.

At some point in history when literacy spread beyond the priesthood, there must have been a similar reckoning. How novel it must have been to consult a “written” “record” halfway through a conversation. Or to converse with the dead in the manner known as reading.

Nowadays, we keep journals, present slide decks, write documentation, publish studies, and do millions of other types of knowledge work that make tacit use of the written word. We do our thinking through the act of writing. The page keeps us honest. We write to explore ideas, be confronted with the vastness of our ignorance, and hopefully think a little more clearly about a certain topic.

So in some sense, we have already become cyborgs. Our cognition is distributed across inorganic components, they just happen to exist outside of our bodies. And once we start to rely on the written word, the pocket calculator, the personal computer, it becomes almost impossible to go without it. Society requires its use.

This combination of the organism and its environment becomes a closed system. As long as we operate in a society where writing, or pocket calculators, or smartphones, or whatever other cognitive aid you can think of are guaranteed to exist and be accessible, the playing field is level.

Now we’re riding the crest of another reckoning with the advent of LLMs. LLMs have made thinking easier. You can offload the mental effort of thinking onto an LLM just by asking it to summarize, explain like I’m 5, tldr, provide 10 variations on a theme, find a synonym, etc. This is fundamentally different from any cognitive aid that came before. Something else is thinking your thoughts for you and putting them back in your mind.

As a result, those who use these tools miss out on the rewards that come from doing the work of thinking. Writing produced with LLMs results in worse recall, less engagement, less creativity, and less perceived ownership. A new MIT study seems to bear this out, lending academic support to something we already knew a long time ago - if you don’t use it, you lose it.

Additionally, LLMs can have a dampening effect on creativity in general. As Vale writes, training data cutoffs and system prompt influence in LLMs end up nudging users to existing patterns and technologies. This is because cutting-edge ideas haven’t made it into the training data yet and may not be established enough for the LLM to mention in its output. So the spectrum of ideas becomes artificially constrained while an overreliance on LLM-provided and LLM-vetted ideas becomes a major hindrance for independent, creative thinking.

But just as before, we could say that as long as we have our closed system, once LLM-”augmented” thinking becomes the norm then the playing field will be level once again and nothing will have changed. It’s impossible to address a claim like that without bringing in notions of absolute good or evil. If we’re reduced to brains in vats being fed a simulation of ice cream on a hot summer day in perpetuity, in what sense is that absolutely better or worse than anything else?

To take a more practical, short-term perspective though, I don’t think this particular system will ever be closed. Even when (if?) we’ve completely incorporated LLMs into all aspects of every day life, interfacing with them will still be orders of magnitude slower than thinking. You wouldn’t pull out your phone in the middle of a conversation to iterate on a witty response to something your friend said. On top of that, until we’re full cyborgs with ChatGPT running locally in our brains, we will always be at the mercy of Windows upgrades, spotty wifi, and power outages.

That means there will always be LLM-free gaps in our days. In our private moments we’ll have no recourse but to confront our own thoughts, if we still remember how. In our inevitable tech-deprived moments of grief and anger, all there will be to help us cope is whatever we’ve read or memorized or written along the way. The path to having a mind that can sustain us in those moments starts now with mindful use of LLMs and the intention to do our thinking for ourselves.

---

---

• ← Intersection types for higher-order functions