In her latest book, Atlas of AI, Kate Crawford maps the tangible and intangible resources needed to create AI and machine learning in order to better understand their social and political effects. Just as maps are specific interpretations of space that are often wrongly seen as objective representations, she unpacks the misconception that AI and machine learning are “bloodless” elements in “purely technical” systems. Instead, she shows how, whether deployed by governments or private businesses, these systems go on to quietly and dramatically shape the world around us. By looking at their material and epistemological origins, she writes, the scars they leave on the earth and people’s lives becomes clear. Paying attention to this is the first step in making better-informed decisions about these systems which will shape our futures. As Hurricane Ida wreaked destruction across the south-eastern United States, the Caribbean and Venezuela, we spoke to Kate over email about the problems with AI and the misconceptions that surround it.
You’ve been working on the topic of AI and automated systems for many years. This space is full of pseudoscience. What is one of the most ridiculous stories that comes to mind?
So much has changed in the 20 years I’ve been working on these issues. But one thing I’ve noticed is how quickly things that seemed ridiculous are being applied in ways that could cause harm. The pandemic has accelerated this phenomenon. A recent review in the British Medical Journal looked at over 200 machine-learning algorithms for diagnosing and predicting outcomes for Covid-19 patients. Some made grandiose claims and sounded very impressive. But the study found that none of them were fit for clinical use – in fact, the authors were concerned that several might have harmed patients. In other cases, things that look fun, like FaceApp, can actually be harvesting images of your face to sell or to train models for facial recognition. So there’s an increasingly fine line between silly and seriously problematic.
Where does the popular understanding of AI systems that are technical systems and therefore somehow objective and neutral come from? What are its effects, and how can or should these perceptions be changed to align with the reality?
It has a long history, all the way back to cybernetics and the early years of AI. Even some of the earliest figures in AI were concerned about the myth of neutrality and objectivity. Joseph Weizenbaum, the man who created ELIZA back in 1964 at MIT was deeply concerned about the “powerful delusional thinking” that artificial intelligence could induce — in both experts and the general public.
Even some of the earliest figures in AI were concerned about the myth of neutrality and objectivity.This phenomenon is now more commonly called “automation bias”. It’s the tendency for humans to accept decisions from automated systems more readily than other humans, on the assumption that they are more objective or accurate, even when they are shown to be wrong. It’s been seen in lots of places, including airplane autopilot systems, intensive care units and nuclear power plants. It continues to influence how people perceive the outputs of AI and undermines the whole idea that having a human in the loop automatically creates forms of accountability and safety.
How should people be thinking about AI? What perspectives can help us to move our discussions beyond the technical achievements of the technology?
There are many ways to think about AI as a technical, cultural and political phenomenon. In Atlas of AI, I look at how AI is becoming an extractive industry of the 21st century – from the raw materials taken out of the earth, to the hidden forms of labour extracted all along the supply chain, to the data extracted from all of us as data subjects. Taking this wider political economy approach can help us see the wider effects of AI beyond the narrow focus on technical innovation. After all, AI is politics all the way down. Rather than being inscrutable and alien, these systems are products of larger social and economic structures with profound material consequences.
How do you explain companies’ obsession with talking about AI ethics, developing framework after framework? How do we move away from this ethics framing?
AI is politics all the way down
As Marietje Schaake wryly observed, there were 128 frameworks for AI ethics in Europe in 2019 alone. These documents are often presented as products of a “wider consensus” but come primarily from economically developed countries, with little representation from Africa, South or Central America, or Central Asia. What’s more, unlike medicine or law, AI has no formal professional governance structure or norms – no agreed-upon definitions and goals for the field or standard protocols for enforcing ethical practice. So tech companies rarely suffer any serious consequences when their ethical principles are violated. Instead, we should focus more on power, an observation that political theorists such as Wendy Brown and Achille Mbembe have been making for many years. AI invariably amplifies and reproduces the forms of power it has been deployed to optimize. Countering that requires centring the interests of the communities most affected, and those who are left out of the usual conversations in technical design and policy making. Instead of glorifying company founders and venture capitalists, we should focus on the experiences of those who are disempowered, discriminated against and harmed by AI systems. That can lead to a very different set of priorities – and the possibility of refusing AI systems in some domains altogether.
You talk about the components which allow AI to exist as being embodied and material – essentially showing that they are the result of different kinds of supply chains being brought together. Why do you think the connections between these intangible, digital systems, the material infrastructure that hosts them, and the people who are affected by them seem so difficult to make?
It’s less obvious, perhaps, because these kinds of connections are intentionally obfuscated. The history of mining, which I address in the book, has always been left at arms’ length from the cities and communities it has enriched.
The description of AI as fundamentally abstract distances it from the energy, labour and capital needed to produce it, and the many different kinds of mining that enable it.Supply chains for information capitalism are extremely hard to research – even for the tech companies that rely on them. When Intel tried to remove conflict minerals from its own supply chain, it took over four years and they had to assess 9,000 suppliers in over 100 countries. I’m glad you mentioned Thea Riofrancos’s work. I’m also influenced by the work of Martín Arboleda. His book, Planetary Mine is great on the way the mining industry has been reorganized into logistical networks and intermingled with information industries. The philosophers Michael Hardt and Antonio Negri call this the “dual operation of abstraction and extraction” in information capitalism: abstracting away the material conditions of production while at the same time extracting more information and resources. The description of AI as fundamentally abstract distances it from the energy, labour and capital needed to produce it, and the many different kinds of mining that enable it. So when we see images of AI, in the press or on an image search, it’ll most often be floating blue numbers, fluffy clouds, white robots and the like, which further abstracts the conversation away from AI’s material and extractive conditions and consequences.
To end on a more positive note! What, to your mind, should come next?
I am inherently an optimist or I wouldn’t be able to keep doing this work. There are so many organizations connecting issues of justice across climate, labour and data. That’s incredibly exciting to see. Of course, we are facing real time pressure now. The IPCC report is just another reminder of why we can’t stall or make minor changes around the edges. Understanding the connections between the computational systems we use and their planetary costs is part of asking different questions, and fundamentally remaking our relationship to the Earth and each other. Or as Achille Mbembe puts it, not only a new imagination of the world, “but an entirely different mapping of the world, a shift from the logic of partition to the logic of sharing.”