Researchers sound alarm on dual-use AI for defense

Defense One. October 22, 2024.

A growing number of Silicon Valley AI companies want to do business with the military, making the case that the time and effort they’ve put into training large neural networks on vast amounts of data could provide the military new capabilities. But a study out today from a group of prominent AI scholars argues that dual-use AI tools would increase the odds of innocent civilians becoming targets due to bad data—and that the tools could easily be gamed by adversaries.

In the paper, published by the AI Now Institute, authors Heidy Khlaaf, Sarah Myers West, and Meredith Whittaker point out the growing interest in commercial AI models—sometimes called foundation models—for military purposes, particularly for intelligence, surveillance, target acquisition, and reconnaissance. They argue that while concerns around AI’s potential for chemical, biological, radiological, and nuclear weapons dominate the conversation, big commercial foundation models already pose underappreciated risks to civilians.

Some companies pitching foundation model-based AI to the military are “proposing commercial models that already have been pre-trained on a set of commercial data. They are not talking about military-exclusive commercial models that have been purely trained on military data,” Khlaaf told Defense One.

The problem lies in how these foundation models are trained: on large amounts of publicly available data, often including personal data that has made its way to the web via data brokers or regular websites and services. “Data is the fundamental issue here. It causes a huge amount of concerns and vulnerabilities, and it’s not being accounted for when we talk about things like traceability broadly,” said Whittaker, who currently serves as president of the Signal Foundation.

These neural nets don’t use reasoning like a human would. Other research has shown they are essentially doing pattern matching on a huge scale, finding combinations of words to create statistically-sound extrapolations of what words might come next, or taking data points on millions of individuals and drawing correlation connections between them (as opposed to causative connections.) Foundation models perform well in scenarios where the cost of false positives is low, such as trying to come up with interesting points for a research paper or a list of people to market a product to.

Applied to military contexts—specifically the task of surveilling a population for potential targets—the cost of a false positive could be an innocent life.

Some militaries, such as Israel, through its Where’s Daddy and Lavender programs, are already employing pattern-extrapolating foundation models to assemble target lists. That’s contributing to the normalization of these tools for targeting in warfare, the authors say.

A report from Israel-based publication +972 found that the error rate for Lavender alone was near 10 percent.

Speaking of Israeli use of AI in its Gaza operations (and beyond) Khlaaf said “We’re seeing… data being pulled from WhatsApp, metadata, right? We’re seeing data pulled from Google Photos and other sources, of course, that we’re unaware of and haven’t really been covered. So the data is inaccurate and [the models] sort of attempt to find patterns that may not exist, like being in a WhatsApp group with a…Hamas member should not qualify you for [a death] sentence. But in this case, that is what we’re observing.”

Said Whittaker, these Israeli tools function based “on surveillance of a given population, right? One of the reasons Ukraine can’t implement systems with the type of characteristics of Lavender and Where’s Daddy is they don’t have that very fine-grain population-level surveillance data, of say, all of Russia’s population. So you see here the role of personal information in informing systems that then use these data models.”

This problem is one reason the Defense Department puts such an

Episode 3120 - October 24 - Tiếng Anh - Các nhà nghiên cứu gióng lên hồi chuông cảnh báo - Vina Technology at AI time

Vina Technology at AI time - Công nghệ Việt Nam thời AI

Episode 3120 - October 24 - Tiếng Anh - Các nhà nghiên cứu gióng lên hồi chuông cảnh báo - Vina Technology at AI time