Sloppy Thinking

Saturday, April 19, 2025 at 10:00am

... In Turing’s famous thought experiment, a human judge is asked to determine which of two correspondents is human, and which is machine. If the judge cannot tell, Turing asserts that the computer should be treated as having essentially achieved the moral and intellectual status of personhood.

Turing’s mistake was that he assumed that the only explanation for a successful computer entrant would be that the computer had become elevated in some way; by becoming smarter, more human. There is another, equally valid explanation of a winning computer, however, which is that the human had become less intelligent, less human-like.

I first discovered Jaron Lanier’s One Half a Manifesto probably over a decade ago, but I’ve returned to it multiple times over the years. I tend to recommend it to anyone interested in software engineering philosophically, with any eye to how computation has affected human cognition and our sense of our place in the world. Lanier intuited, more than two decades prior, the metastacization of certain negative thought patterns in Silicon Valley into what we today refer to as “tech bro culture.”

The chunk I’ve quoted above is particularly relevant today with regard to the cluster of technology patterns commonly referred to as ”artificial intelligence.” While I’ve enjoyed a recent opportunity to learn many details about various implementations of this cluster of technologies (and have had quite a lot of fun doing it), I dislike this nomenclature. ”Artificial intelligence” should properly refer to algorithms, software systems, robots, or machines that become sentient—truly self-aware. This cluster of technologies can generate some impressive outputs, but I think it naive to assume sentience, or even the nearness of sentience.

The much more interesting and pressing concern is how these technologies, which I’ll refer to as large languages models (LLMs), are affecting how humans think and behave. This is a much more pressing concern.

As a software engineer myself, I cannot help but notice how LLMs have affected the profession. Firms such as Meta and Salesforce have openly engaged in the diminution of their engineering workforce, admitting that they want their fewer remaining engineers to make up for the productivity of their lost human colleagues by utilizing LLM tools in constructing code.

My own work recently has given me the opportunity to use Microsoft Copilot, an LLM-based tool that predicts what code I might like to type next. As is typical for these kinds of tools for human language generation purposes, the more formulaic or generic my problem space is, the better the results. To give a concrete example, Copilot is excellent at writing unit tests for me. It is utterly useless at predicting production code for a brand new feature, even with excessive prompting.

This brings me to “vibe coding,” an up-and-coming software practice of engineers (and even companies who support such things) in which engineers attempt to construct nearly all their code by supplying prompts to LLMs. The tactic on the companies’ part is pretty clear. The hope is that eventually product managers and executives will be able to “vibe” their way to successful products without the need for large, specially-trained (and expensive) engineering teams.

This brings me back to Jaron Lanier’s article. Just as Turing, for all his insight about early computer architecture, forgot, in his famous axiom, about the human component in his test, so today are the companies that are betting on a future of engineer-less code generation forgetting that the primary operator of an LLM is a human being, and that getting good results from an LLM has as much to do with the technology generating the response as the human generating the input.

LLMs are an exciting new technology. It makes sense to invest in them, particularly with regards to code. But let’s not fool ourselves about where code ultimately comes from. If you eventually take away the humans whose intuition and experience grant them the ability to generate the correct insight for a given problem (crucial for cranking out code and prompts both), then such companies will ultimately end up with broken systems that they are unable to repair. And the cost of hiring engineers, after demonstrating callousness toward the profession rather than care, will be quite high.