Consumer and Supplier - The self serving feedback loop
I recently attended the UN informal exchanges on AI in the military domain in Geneva. While I was there to discuss the current research project I am leading on decommissioning military AI systems, I noted one particular topic which surfaced consistently across the three days - the relationship between States and Industry.
Across the three days, discussions surfaced on the role of industry in military AI and what interventions are needed at this level to ensure the systems they are developing are safe, responsible and lawful. What I felt was missing from the discussion was emphasis on the role of States.
Industry is ultimately a supplier to a very willing consumer - militaries. They are developing products to meet a demand that is set by militaries. If there was no demand, there would be no product. We have long passed the point of innovation for the betterment of society, albeit I contend this could be argued to never have been the case. Current technical “innovation” is motivated by fiscal gain, and there has always been money in war.
Militaries around the world are jumping on the “AI peacocking” bandwagon, racing to acquire inadequate systems that are not yet technically sufficient for the military operations they’re intended for. If States are willing to rapidly acquire a sub-tier product, why would industry say no? What for-profit organisation is going to reject millions of dollars on the basis of their product not being good enough? Let’s not forget, Silicon Valley was built on embellished narratives and false promises. It’s the old adage of build it and they shall come, but in the case of Silicon Valley it’s more along the lines of pretend you can build it and they shall come.
But the root of the problem here is not in those who build, or pretend to build, but in those who choose to come. AI would not be the “success” it is today if their wasn’t a willing audience and consumer. I use the word success here cautiously. If you measure success in monetary value then the AI industry is bountifully successful; however, if you measure it by any other means, the argument of success doesn’t hold as true.
We can continue to debate the ethics of technology companies, but the fact of the matter is it is not in their fiscal interest to make different decisions. If we want change, we need to incentivise them to change, and that incentive needs to be financial.
Across Australia and New Zealand, cars are accredited with what’s known as an ANCAP safety rating - the Australasian New Car Assessment Program. This is a voluntary assessment which rates cars on a safety scale of 1-5 stars. Despite being a voluntary assessment, all cars sold across Australia and New Zealand undertake this assessment because it is financially beneficial to do so. The ANCAP safety rating is ultimately a marketing scheme. Consumers see 5 stars for safety on one car and no stars on a competing vehicle and are convinced the one with the stars must be safer. The rating system is a signposted incentive for consumers to make the “safer” choice. In this scenario, the power sits within the choice of the consumer, whereby their purchasing decisions have incentivised the supplier - car manufacturers - to opt into a voluntary safety rating program. A rating program which costs them time and money to adhere to.*
In the case of military AI, the power for change also sits within the choices of the consumer. States are subject to obligations under international humanitarian law (IHL) and the laws of armed conflict (LOAC). There is open debate about the applicability of these obligations to military AI systems; however, these obligations are technology agnostic and apply unequivocally, irrespective of the technology used, making the debate somewhat superfluous. Honestly, I think the debate is a scapegoat to clear the path for rapid adoption and deployment. While there is room for nuanced discussion on applicability and the potential need for additions to existing obligations, overall, the idea of existing obligations being entirely inapplicable to AI systems simply does not hold.
At the moment, the primary incentive for States seems to be military effectiveness, with AI posited as the singular means for achieving this - this was not explicitly stated during the informal exchanges. Many States made notes of “opportunities” and “benefits” of AI but there was somewhat of a consensus on also managing the risks. My comment stems from public discourse beyond the informal exchanges, particularly the U.S. DoD AI Acceleration Strategy released in January this year. The topic of “AI solutions” was also quite prominent during the REAIM Summit held in February this year, which I reflected on in a piece I wrote for Just Security.
The relationship between the tech industry and militaries - the supplier and the consumer - is currently a self serving feedback loop that needs external intervention. The tech industry marketed a magical product that solves all military problems, including those we didn’t know existed; militaries latched on to this and became a willing and enthusiastic consumer; the tech industry continued to mature this product to provide more “solutions”; militaries continued to purchase those “solutions”.
A blogger who’s work I really enjoy, Baldur Bjarnason**, wrote a blog post which talks about the idea of a technopoly, which he describes as a state in which “the only ideas and thoughts that have social and cultural legitimacy are those that support, are supported by, and are mediated through technology”. The whole premise of this idea is that technology is itself a cultural practice. It’s not about how good or bad technologies are, it’s about how well they support technology as a culture.
If we look at AI, a 2025 MIT study found 95% of organisations received a zero return on investment in generative AI. Despite this, the generative AI market is expected to grow beyond $1 trillion over the next ten years. How remarkable to see such a projection for a technology that is yet to deliver on its promises. How much of that expected market growth can be attributed to the Defence sector is not clear, but one would imagine it would count for a notable percentage. If current trends are anything to go by, it seems the trajectory of investment into military AI will continue to increase in the near future, despite the glaring issues that come with the use of this technology in military operations. Which brings us back to the central issue of who is responsible for addressing these issues, tech companies or States?
I don’t fully agree with the premise of private technology companies bearing the responsibility of ensuring safety and legal obligations of military AI systems are met. Tech companies are not going to take it upon themselves to meet obligations that are not currently being enforced by States because it is simply not in their fiscal interest to do so. If the consumer is willing to purchase AI systems without the assurance of meeting safety and legal obligations, why would tech companies disrupt that status quo? Assurance of any kind costs time and money - this is why there is such a strong push against regulation under the guise of stifling innovation. Regulation does not stifle innovation, it can however stifle profit margins.
The power for change sits in the hands of the consumer. Unless a consumer shifts the benchmarks of demand, tech companies will continue to sell sub-tier systems that lack technical integrity. For other industries, regulations have helped to shift those benchmarks, with regulators being an external point of disruption between the supplier and consumer. This is what the Defence industry needs. The self serving feedback loop between the supplier (tech industry) and the consumer (militaries) needs external intervention to disrupt the current status quo.
What that external intervention should look like is an open question that I don’t have the answer to. But what I do know is that there is currently a missing piece of the puzzle and without that piece, the feedback loop between the tech industry and States will continue its self serving cycle, at the expense of effective and robust technical systems.
*****
Acknowledgements
*Full credit to my colleague Hanna Kurniawati who used the ANCAP safety rating example when speaking on a panel I moderated a few years back. Her comments on this panel are what I have reflected on in this section of this blog post.
**I first discovered Bjorn Bjarnason’s blog through my friend Kathy Reid who shared one of his pieces on LinkedIn. Kathy is a great source of reading recommendations, giver her a follow if you’re interested in tech related content.
