More countries are incorporating artificial intelligence into their military structures. The technology is being used to analyze large volumes of data and select targets, in logistics and intelligence, and to help with decision-making. Companies developing AI systems for the military say the spread of artificial intelligence in the defense sphere is inevitable. But the consequences of AI’s arrival on the battlefield are determined by military officials themselves — through their choices of suppliers, systems, and tasks. People, however, tend to overestimate the capabilities of any new technology. In this case, that could lead to rising global tensions and critical errors caused by system failures. Oversight during the development of military AI systems could help, as could training service members and their commanders to critically assess the systems’ responses, writes Jacquelyn Schneider, Director of the Hoover Wargaming and Crisis Simulation Initiative (Hoover Institution), and an affiliate with Stanford's Center for International Security and Cooperation.
AI seems to be on an inevitable march into conflict. Media reports claim Israel used AI tools Gospel and Lavender to sift through massive amounts of intelligence and derive target lists in Gaza. Meanwhile, the U.S. is integrating AI into logistics and intelligence, including Palantir’s Maven Smart System, which uses AI to monitor satellites in space, task drones, and provide decision support for commanders. Future uses include AI-operated autonomous drone swarms and potentially even AI-enabled mission command on the modern battlefield.
AI is no longer the future of war. Instead, it is part and parcel of any modern conflict. Certainly, the companies building these AI platforms view their growth as unescapable. As Palantir, one of the leading AI providers to the U.S. Department of Defense, asserted, “The question is not whether A.I. weapons will be built; it is who will build them and for what purpose.”
The question is not whether A.I. weapons will be built; it is who will build them and for what purpose
U.S. Secretary of Defense Pete Hegseth and Claude LLM developer Anthropic are currently involved in discussions that will have wide-ranging consequences. Decisions made by militaries — about what developers they use, the algorithms they rely on, and how their military personnel integrate the systems — will determine whether AI leads to more or less instability. AI can create remarkable advantages on the battlefield, enable logistics and mobilization, and augment early warning. It may also help make war more discriminate and decrease the danger of emotions like fear, anger, or even fatigue, all of which can lead to violations of the laws of armed conflict. In short, the choices made by militaries and AI developers today will shape whether states can harness the power of AI for deterrence and stability or whether the technology will end up driving states to war.
Confidence and overconfidence, technology, and war
Perhaps the greatest danger of any emerging technology is that new technologies shift the balance of power so profoundly that states believe they either have an advantage or are so inherently vulnerable that they must attack. While technologies rarely create immediate, enduring advantages that fundamentally change who can take and hold territory, the lure of the possibility is devastatingly powerful. Confidence, and overconfidence, in what an emerging technology can do on the battlefield can make states more likely to launch preemptive strikes, undertake offensive campaigns, or fail to anticipate how a technology might go awry. Indeed, in research I conducted on cyber vulnerabilities and conflict, I found that players’ routine overestimation of offensive cyber capabilities in wargames led to a greater chance of first strikes on nuclear arsenals.
While technologies rarely create immediate, enduring advantages that fundamentally change who can take and hold territory, the lure of the possibility is devastatingly powerful
The overconfidence technology can inspire when it comes to deciding whether it is worth going on the attack is also exacerbated under certain psychological conditions. The Dunning-Kruger effect, for example, describes a phenomenon in which individuals with less skills or abilities tend to overestimate their capabilities. They don’t know what they don’t know, making them overconfident in tasks in which they have little expertise. As the authors explain, “people who are unskilled in these domains suffer a dual burden: not only do these people reach erroneous conclusions and make unfortunate choices, but their incompetence robs them of the metacognitive ability to realize it.”
This tendency to overestimate a technology’s capability can likewise come to the fore when decision-makers are faced with situations that have no easy solution, and also when the technology seems to support decisions they already wanted to make. For example, in the cyber wargame I referenced earlier, players were faced with a no-win situation: put on the brink of nuclear conflict, with no clear way of either de-escalating or defeating the enemy. When given a new cyber capability to attack the enemy’s nuclear command, control, and communications, players were willing to suspend disbelief about the technology’s feasibility. Even though the game told players that it wasn’t clear what the scope and duration of the effects of the cyber exploit could be, players wished away the uncertainty and leaned into the technology in the hopes of getting out of the tricky situation. This phenomenon was most powerful with players who had limited knowledge or training with nuclear strategy, and we found that those with limited nuclear expertise had a greater tendency to resort to use of the cyber exploit and attacks on enemy nuclear arsenals.
The AI bubble: Are we overconfident in AI?
Does AI imbue overconfidence? Emerging research suggests this is a possibility, and experiments with young teachers and inexperienced investors suggest that the Dunning-Kruger effect is particularly powerful with AI. Further, many AI agents are designed specifically to engender confidence in their users, leading AI agents to both underestimate the uncertainty of their own recommendations and to intentionally obfuscate that uncertainty to users. This leads to overconfident AI models and agents that then overconfidently present information to users who may — based on experience, beliefs, or personality — already be primed to put more faith in the technology than they perhaps should.
This confidence in the power of technology is concerning because, despite extraordinary progress with AI, there are fundamental concerns about the safety and reliability of existing AI systems. Research has found that AI itself can be overconfident when faced with uncertainty — when placed in a game of Go, the AI agent took risky offensive plays with sometimes catastrophic results. Further, wargames played with AI agents (as compared with to human players) demonstrated not only a pattern of escalation, but also the fragility of the models and tendency to hallucination when asked to iterate over time.
Despite extraordinary progress with AI, there are fundamental concerns about the safety and reliability of existing AI systems
The limitations of AI in lab and non-conflict environments will become dangerous vulnerabilities in war when models, training data, and users will be targeted, manipulated, and deceived. Users, inexperienced with interrogating AI, may find themselves in a scenario of binary trust in the AI tools they rely on to fight war. Trusting everything the machine is telling a user may lead to overconfidence, accidents, and inadvertent pathways to war. On the other hand, however, trusting nothing will leave militaries dead in the water, unable to respond when they don’t know what they can believe from AI tools.
This will be especially dangerous for young military personnel — new to combat, primed for the Dunning-Kruger Effect, and less likely to push back against AI errors in complicated military systems. Similarly, new decision-makers without significant military experience will be vulnerable to the Dunning-Kruger Effect and therefore may be more confident in AI assessments that present optimistic beliefs about the ease of victory. The applicability of the Dunning-Kruger Effect to these important populations demonstrate how even with the inclusion of “human control” in the most dangerous and violent weapons decisions — like the choice to go nuclear — young missile silo operators or new commanders-in-chief may fail to properly understand the uncertainty of AI.
Overconfidence, the machine, and authoritarian regimes
Is this overconfidence in AI more likely to occur in some types of regimes than others? And how might that manifest on the battlefield? Scholarship on the impact of regime type on military effectiveness finds that authoritarian regimes are less likely to trust their militaries and, therefore, they build less capable militaries. This occurs for a variety of reasons but much of the phenomenon stems back to trust and how a lack of trust in military subordinates leads authoritarian regimes to consolidate control at the highest level, making their forces less resilient and adaptive.
The tendency for authoritarian regimes to distrust their military leaders has significant implications for AI. If a political leader does not believe their subordinates will comply with orders, they may be more likely to use loyal AI agents as a substitute for humans. Those AI agents, designed to defer and support the regime, will likely exaggerate sycophantic tendencies and buttress authoritarian leaders’ desired policies. Further, regimes can invest in AI agents developed to optimize regime goals without ethical concerns, engineering out human emotions like empathy, disgust, or horror and thereby remove some of the most important factors limiting the use of violence. Imagine, for example, a deadhand AI developed not just as a response to an adversary’s first strike, but also as a way for leaders to ensure that human subordinates will push the button in situations when all minimum conditions doing so are present.
The biggest danger of AI is humans
Like many technologies, the danger of AI is inherently human: how we use and develop machines as agents of our worst tendencies. After the horrors of World War I, scholars tried to make sense of how a war that made no sense could have killed millions. One of the early explanations was technology — that railroads, telegraphs, and steam had built arsenals that necessitated a war no one wanted. However, despite the desire to defer guilt to the machines, it was humans who made the choices to build those arsenals. It was great men like Kaiser Wilhelm who believed they could use those technologies to win a war, the costs of which they significantly underestimated.
The danger of AI is inherently human: how we use and develop machines as agents of our worst tendencies
Today, humans will make decisions about how we build AI arsenals, how we use those arsenals in war, and what role they play in the most important decisions — including the decision to embark on nuclear war. Teaching decision-makers about the limitations of the technology and training users to interrogate AI will decrease the dangers of overconfidence, while prioritizing safety within development will decrease the possibility of accidental failures. Together, human agency over the inevitability of the technology can steer its adoption to safer integration of that technology in war.
