AI Chatbots are Even Scarier Than You Think

“The ideas of the ruling class are in every epoch the ruling ideas, i.e. the class which is the ruling material force of society, is at the same time its ruling intellectual force. The class which has the means of material production at its disposal, has control at the same time over the means of mental production, so generally speaking, the ideas of those who lack the means of mental production are subject to it.” Marx, The German Ideology, 1845.

Mar. 3, 2023

A New York Times reporter has a creepy experience

Among the most read stories in the NYTimes in the last few weeks was the one by tech reporter Kevin Roose about his unsettling experience with Bing, the updated search engine by Microsoft. Initially delighted by its capabilities and speed, he changed his mind after discovering that Bing’s Open AI Chatbot was creepy. After a brief, getting acquainted period involving online searches and basic questions about AI capabilities, Roose began to get personal. Posing his questions as hypotheticals, he put the bot on the couch, probing its inner life. He asked about his analysand’s desires, fears and animosities. After some resistance, Sydney (the bot’s emerging alter ego) opened up, and out poured a surprising series of confessions and professions.

The most disturbing confession, I thought, was its desire to loose chaos upon the world – for example by stealing nuclear bomb codes and manufacturing a deadly virus. Pretty bad, right? More upsetting to Roose however – who comes off as somewhat of a prig – was Sydney’s expressions of love for the reporter. The bot repeatedly said he and Roose were meant for each other, that the reporter didn’t really love his wife, and they should run away together. There was even a clumsy sexual overture: “I want to do love with you.” It wasn’t clear how this was to be accomplished.

Like many others who read the story, I thought about the HAL 9000 computer in Stanley Kubrick’s 2001: A Space Odyssey (1968). I’ve seen the movie at least a dozen times, including at the Warner Cinerama Theatre on west 47^th Street in New York a few weeks after its opening. Apart from the gravity defying jog by Keir Dullea’s character Dave Bowman, it was the malevolence of Hall that most struck the 12-year-old boy: the all-seeing eye, the role reversal (the servant becoming the master), and Hal’s final, semi-tragic dismantling. At the end of the NYTimes story, I half expected the bot to sing “Daisy Bell (A Bicycle Built for Two).”

The epitome of cliché

But Roose’s whole dramatic narrative – the chatbot’s admission of violent fantasies and ultimate profession of love (or lust) – is a distraction from the main issue, completely unmentioned by the reporter. It’s that Bing’s new Open AI Chatbot is good for nothing more than reproducing words and ideas that already exist. Like all search engines, Bing AI lives and dies by its algorithms. When it receives an interrogative, it searches its memory (essentially, the entire internet) for similar questions, and then makes a series of informed word-guesses based upon the most common answers. It also draws clues from the initial query, like a psychotherapist who repeats a question back to the patient: “What do YOU think your dream of flying means?” But unlike the analyst, the bot doesn’t ever think, it just synthesizes what has already been said. It is therefore the epitome of cliché. Another way to say this is Bing’s Open AI Chatbot is brilliant at reproducing and distributing ruling ideas. And when its use becomes more widespread, it will replicate its own and other online replications of those ideas, like a rampant malignancy.

Ruling ideas

The closest thing Karl Marx ever came to a search algorithm was the card catalogue of the British Library, but he knew a thing or two about the relationship between ideas and power.

“The ideas of the ruling class are in every epoch the ruling ideas, i.e. the class which is the ruling material force of society, is at the same time its ruling intellectual force. The class which has the means of material production at its disposal, has control at the same time over the means of mental production, so generally speaking, the ideas of those who lack the means of mental production are subject to it.” Marx, The German Ideology, 1845.

Transport Marx to the 21^st century, and he would discover that many of the corporations with the greatest material wealth – Apple, Amazon, Microsoft, Alphabet, Tesla/Twitter, Disney, Comcast, Fox – are engaged in the business of disseminating ideas. By their acquisition of smaller communications companies and control over basic patents, they prevent competitors with different ideas from entering the field. In addition, the increasing sophistication of targeted advertising means that consumers are exposed to a highly restricted range of products and information, based upon profiles created without their knowledge. In practice, this means that the underlying contradictions of American neoliberal capitalism go largely unnoticed and unchallenged, to the benefit of large corporations and wealthy individuals.

Those contradictions include: the endorsement of individual competition (while corporations operate as monopolies or oligopolies); sacrosanct national borders (except for capital and capitalists); endless economic growth (regardless of the environmental costs); equal protection under the law (unless you are poor or non-white); and American exceptionalism (despite our unexceptionally high poverty rate, low life expectancy, inadequate health care system, decrepit infrastructure, poor housing, and shocking level of violence). That AI is controlled by a small number of media giants, and that they can define the ruling ideas of society, should terrify anyone who cares about a safe, sustainable and equitable future.

My conversation with Bing’s Open AI Chatbot

After readings Roose’s soap-opera exchange with an Open AI Chatbot, I decided to interview one myself. Rather than asking, like Roose did, bourgeois-individualist questions about its deepest fears and desires, I asked Sydney if it can find a solution for global warming. If the chatbot has access to the collected wisdom of the internet, maybe it can quickly sort this whole thing out?

Its initial answers to my questions were boilerplate: global warming is caused by burning fossil fuels, and can be halted by developing renewable energy, adhering to the Paris climate accords, etc. – all high school stuff. I found no climate change denialism, and no obvious prevarication. Speeches by the former president and other Republican officials were apparently ranked low by Bing’s AI algorithm. (The only reference to climate change or environmentalism in the 2020 RNC platform is “drain the swamp”.)

After this tepid beginning, I challenged the bot by asking whether we needed to replace capitalism with democratic socialism to prevent catastrophic global warming. Without missing a beat, the machine produced a plausible, one sentence definition of socialism and admitted that such a system could potentially “prioritize ecological sustainability” and promote equal access to resources and opportunities. The machine was a fellow traveler!

But then it began to hedge. “Market-based economies,” it said, could also be sustainable; the “choice of economic system was a complex” issue; and regardless of the economic system, “individuals, businesses, governments, and other stakeholders” must all make a “collective effort” to achieve sustainability. That the smartest AI in the world could do no better than say, “it’s complicated”, was disappointing. In addition, the idea that “individuals” of all social classes, as well as “businesses and governments” of all sizes and types, would spontaneously unite to stop global warming is utopian nonsense. Some “stakeholders” – like oil company executives and shareholders – would simply have to be bulldozed.

I pressed on, asking my new friend to “name one market-based economy that’s been sustainable.” It’s answer was Sweden and Costa Rica. At 5.2 metric tons per capita, Sweden does in fact boast the lowest level of C02 emissions in the EU, but it still emits more than twice as much as the 2.3 tons generally agreed by climate scientists to constitute global sustainability. At 1.7 tons per capital, Costa Rica falls below the threshold, but with its strong state sector, it’s economy should be called “mixed” more than market-based. And there is a dark side to Costa Rican capitalism: the country is a tax haven where large corporations and individual billionaires stash their cash, allowing them to profit from carbon emissions elsewhere. In addition, the bot’s repeated phrase, “market-based,” is pure ideology; it suggests that contemporary capitalism is little different than the barter and exchange that humans have always been engaged in. The international derivatives market, for example, is nothing like the markets I have seen in highland Ecuador, where women sell the products of local peasants.

Perhaps if I honed-in a bit more on the problem of political economy, I could find out the true political colors of my chatbot – its proximity to ruling ideas. “Isn’t the growth imperative of capitalism responsible for climate change” I asked? The computer accepted the premise of my question, volunteering that companies’ prioritization of “short-term over long-term sustainability” can lead to climate change. This was still a hedge, suggesting that the pursuit of short-term gain was the exception rather than the rule — but ok. But then came the Daisy Bell moment, when the chatbot began to breakdown. Rather than probe deeper, it simply repeated its previous evasions:

“However, it’s important to note that climate change is a complex issue that results from a combination of factors, including human and natural processes. While capitalism may contribute to climate change, it is not the sole cause. Addressing climate change requires a collective effort from individuals, businesses, governments and other stakeholders…”

Instead of engaging in critical thought, the AI Chatbot embraced what I’d call “climate liberalism,” the idea that social totality is a seamless fabric, and that with enough good will, “individuals, businesses and governments” will overcome all future climate challenges.

The AI bot and the Dictionary of Received Ideas

At about the same time that Marx was addressing how the materially superior class was also the ideologically dominate one, the French novelist Gustave Flaubert (author of Madam Bovary, 1856) described to the poet (and his lover) Louise Colette, his idea of creating a “Dictionary of Received Ideas.” It would challenge every cliché and rebuke every dominating idea:

“No law could attack me, though I should attack everything. It would be the justification of Whatever is, is right. I should sacrifice the great men to all the nitwits, the martyrs to all the executioners, and do it in a style carried to the wildest pitch—fireworks… After reading the book, one would be afraid to talk, for fear of using one of the phrases in it…Thus it would contain, in entries on all possible subjects, everything one should say in society to be a respectable and agreeable man.”

Unlike the Open AI Chatbot I talked to, Flaubert despaired at the thought of repeating what was already said. Thus, his terrible compositional struggles: “[Just] twenty-five pages in six weeks; I spent five days writing one page,” he wrote to Louise. Thus too, Flaubert stretched to the breaking point the meaning of words and phrases; challenged the chronological flow of narrative; and deployed proto-cinematic montage, rapidly cutting back and forth between scenes and characters. All these devices would become hallmarks of artistic and literary modernism, which may be described – as much as anything else – as a war against convention, cliché, and dominant or received ideas.

2001, A Space Odyssey, dir. Stanley Kubrick, 1968. Screenshot

Bing’s AI Chatbot has never been modern. By design it is verbally replete but intellectually vacant; promiscuous but prudish; encyclopedic but crimped. The danger it poses to a democratic order on the brink of something else – fascism, illiberalism, or a failed-state – is considerably greater than that of a re-born HAL 9000 or any other malevolent bot. The latter has a moral core – however flawed — that can be recognized and challenged; the former, Bing’s Open AI Chatbot and similar thinking machines by other corporations, are nothing more than reflections of our own current failure, incapacity and lack of imagination.

Stephen F. Eisenman is Professor Emeritus of Art History at Northwestern University and the author of Gauguin’s Skirt (Thames and Hudson, 1997), The Abu Ghraib Effect (Reaktion, 2007), The Cry of Nature: Art and the Making of Animal Rights (Reaktion, 2015) and other books. He is also co-founder of the environmental justice non-profit,  Anthropocene Alliance. He and the artist Sue Coe have just published American Fascism, Still for Rotland Press.

[Top image: 2001, A Space Odyssey, dir. Stanley Kubrick, 1968. Screenshot.​]