“ What degree of human involvement should there be in the use of force? ” Paul Scharre on weapon autonomy and the human role in future warfighting

16 Pages·2021·1.849 MB·English

by  Paul Scharre, Metis Institute for Strategy and Foresight Bundeswehr University Munich (ed.)

Interview
“ What degree of human involvement
should there be in the use of force? ”
Paul Scharre on weapon autonomy and
the human role in future warfighting
No. 4 | April 2021
The views expressed in Metis Interviews are those of the interviewees. They do not reflect the opinion
of the Bundeswehr, the Federal Ministry of Defence, or the Bundeswehr University Munich.

MMeettiiss IInntteerrvviieeww || NNoo.. 44
HHuummaann mmaacchhiinnee iinntteerraaccttiioonn –– PPaauull SScchhaarrrree
Interview
my
Paul Scharre on weapon autonomy and US Ar
o:
the human role in future warfighting ot
h
P
g |
or
dia.
e
p
wiki
urce:
o
S
Fig. 1 British soldiers with captured German Goliath remote-controlled demolition vehicles (Normandy, 1944).
A fter multiple tours to Iran and Afghanistan as an Army Ranger, Paul Scharre
played a key role in establishing policies on emerging weapons technologies at
the US Department of Defense. In particular, he led the working group that drafted
Directive 3000.09, the Pentagon’s 2012 policy on autonomy in weapon systems.
With “Army of None”, Scharre also published a widely popular book on a utonomy in
weapons systems. The interview was conducted by Metis Head of Research Dr. Frank Sauer
in March 2021 via Zoom. This transcript was slightly redacted.
Sauer The alternative, much simpler, at the United Nations (UN) where
Getting a conceptual handle on functionalist approach is to define you see people heatedly disagreeing
what weapon autonomy is seems weapon autonomy as nothing more about autonomous systems. And then
to be less of an issue than it used but the selection and engagement you listen to what they’re saying – and
to be. But it remains important to of targets without human inter- they’re talking about d ifferent things!
first make clear what it is that we vention. I have a strong suspicion The term autonomous weapons
are talking about. Because there which one of these two viewpoints conjures a wide variety of visions in
still are two competing approaches. you choose to adopt. But, to get people’s heads. When some people
The first approach is often cou- our conversation started, let me say autonomous weapons, they are
pled to the effort of delineating ask: how do you conceptualize in fact envisioning the Terminator.
between “old” – automatic – and weapon autonomy – and why? You can’t get through a discussion
“new” – autonomous – weapons about autonomous weapons with-
and to define the latter as a cat- Scharre out that coming up. Other people
egory of weapons characterized This is a topic that I have strong feel- are thinking of something much sim-
by specific new capabilities such ings about. The terminology is chal- pler: a ‘ Roomba’ with a gun on it, ef-
as assessing a situation, making lenging. If you put ten engineers in a fectively. And I would like to say that
decisions, and learning, almost room and you ask them to define au- both of those are problematic, but for
like a semi sentient machine tonomy, you’re going to get a dozen very different kinds of reasons. They
attacking humans – notice how different answers. I’ve certainly been are probably both a bad idea, but
I am not saying Terminator? a part of discussions internationally there are problems that come from
3 / 16
Metis Interview | No. 4
Human machine interaction – Paul Scharre
to refer to goal-driven sys- This is why I definitely come down
tems. Take an autonomous on the side of a functionalist defi-
car as an example. It is still, nition of autonomous weapons. In-
of course, obeying its pro- stead of talking about the supposed
My personal view gramming, it’s not engag- “level of intelligence” of the machine,
ing in free will or becoming we are really talking about what the
is that trying to alive, but it has more ca- functions are that it is performing.
pacity to take inputs from In particular, we must ask if it is per-
make distinctions its environment and make forming the functions of searching
“decisions”. That is a load- for and selecting targets to attack
about definitions ed word when it comes on its own. Here, I prefer to envision
to machines. But it means a decision-cycle according to which
for autonomous that the machine is making an autonomous weapon would be
choices based on its pro- one able to complete that entire
weapons along gramming guidance and cycle on its own: to sense the envi-
environmental inputs in ronment, identify targets, make de-
this spectrum order to accomplish a goal. cision whether or not to attack and
My personal view is that carry out the engagement. These
of “intelligence of trying to make distinctions are obviously weapons still built by
about definitions for auton- humans; they’re still designed by hu-
machines” is largely omous weapons along this mans. We’re not talking about robots
spectrum of “intelligence building robots on their own and
a fruitless exercise. of machines” is largely a running amok. Instead, this would
fruitless exercise. I’ve ob- be weapons that would be launched
served that people tend to by humans and put into the bat-
use the word “autonomous” tlefield environment for some pur-
to refer to things that don’t pose. So, there is, at a broader level,
exist yet or that seem a bit still human involvement. But that
mysterious. And then, once involvement is changed in signifi-
we build them and you cant ways that is certainly worthy of
autonomous systems that are not are able to look under the hood, the discussion. And we have seen this
smart enough, and then there are same people say: “Well it’s not really unfolding internationally over the
problems that might come from sys- autonomous, it’s just automated, it’s last couple of years, as various ac-
tems that are very sophisticated. following this programme”. All these tors have been very engaged in the
So, I think this distinction between things are following the programme! question if it is a good idea to begin
automatic, automated and autono- We’re not building machines that are crossing that line to build autono-
mous is real in the sense that people coming alive! mous weapons.
use those terms to refer to different
levels of sophistication of
machines. They refer to
something very simple as
automatic – something
that might have a very clear
linkage between input and I definitely come down on the
output. A tripwire is an ex-
ample. Next, they would side of a functionalist d efinition
use the word automated
to refer to more complex of autonomous weapons.
systems – something like
a programmable thermo-
stat in your home. You set
in some parameters, and
then it performs a func-
tion. Finally, they tend to
use the term autonomous
4 / 16
Metis Interview | No. 4
Human machine interaction – Paul Scharre
Sauer when we think about autonomous in robotic platforms, in weapons sys-
So if we focus on the functionality weapons today. tems, in defensive systems. Because
of the system – or the “system of there are many, many advantages. At
systems” because functionality Sauer the same time, I am hard pressed to
might be distributed, that is, the Why are militaries pursuing point to development programmes
sensor and the shooter platform weapon autonomy? that are clearly designed to cross
might not necessarily be the same Could you run us through the the threshold to fully autonomous
asset – then the discussion about main advantages of having weapons that would be targeting
weapon autonomy becomes one increased weapon autonomy? on their own. There are some exam-
about how to shape that func- ples of things that have already been
tionality and human involvement Scharre around, like the Israeli Harpy loitering
rather than about defining a Let’s be clear, the main advantage munition, and there are certainly at
distinct category of weapons? is military effectiveness. That’s why least thirty countries that have super-
militaries are interested in investing vised autonomous weapons systems
Scharre in increased autonomy in weapons for defensive purposes like Aegis
Yes, I 100 % agree with that. And in- systems. And I want to draw attention or Patriot or even active protection
deed, that is a great point: it need to the fact that I am using the term systems for ground vehicles to shoot
not be a single platform or weapon “increased autonomy in weapons sys- down incoming rockets. But currently,
that we’re talking about! It could be tems” – which is a bit fuzzy and am- while a lot of the development pro-
a system that is distributed. And that biguous – rather than “autonomous grammes look like they are heading
is what makes it trickier. The decision weapons” per se. Because when I down a path towards greater autono-
could be distributed as well. look around the world at the invest- my, it’s either unclear how far they are
There are situations today where ments that militaries are making, we going to go and whether they go to
you have a single person looking see every major military investing in “full autonomy” or if they tend to walk
down the barrel of a rifle, making a increased autonomous functionality up to the line but perhaps not cross it.
decision about whether to
use lethal force. But then
there are other situations. If
you look at a modern mili-
tary launching a cruise mis-
sile from a ship – that use
of lethal force decision is
quite distributed. You have
targeteers planning out
various targets and making
decisions, then those get
approved, they get on the
target list. Then there’s a
decision to actually launch
the strike, then you have
people on the ship that are
planning out the strike and
a commander who author-
izes the launch. And in the
end, it might be somebody
else who is actually push-
ing the button to launch
the weapon. So, when you
ask who is responsible,
then there are still people
responsible today. But it
may not be one single per-
son. That authority may be
distributed – which defi-
nitely complicates the issue Fig. 2 IAI Harop (or IAI Harpy 2) at Paris Air Show 2013. | Source: wikipedia.org, Photo: Julian Herzog
5 / 16
Metis Interview | No. 4
Human machine interaction – Paul Scharre
Fig. 3 "A Long Range Anti-Ship Missile (LRASM) launches from an Air Force B-1B Lancer during flight testing in August 2013." | Source: wikipedia.org, Photo: DARPA
Metis Interview | No. 4
Human machine interaction – Paul Scharre
Metis Interview | No. 4
Human machine interaction – Paul Scharre
Sauer the Campaign to Stop Killer Robots, principles is going to be less convinc-
Let’s talk about this “line” and no Human Rights Watch, Article 36 and ing with Vladimir Putin or Xi Jinping.
one seeming particularly keen to many others. They put this issue on Appealing to concerns about strate-
cross it. There is a lot of debate the agenda internationally in the UN. gic ability may be an area where there
about the legal implications, the And this really put the humanitarian might be some areas of agreement.
ethical risks, the political risks disarmament concerns front and cen-
of operating weapons auton- tre. Just by virtue of attention, issues Sauer
omously. 1 Which one of those surrounding strategic considerations If you had the power to wave
weighs the heaviest on your and the risk of instability among a magic wand in light of these
mind in terms of where weapon states, as you and Jürgen Altmann, strategic risks and draw specific
autonomy could go wrong? Jean-Marc Rickli, Mark Gubrud and lines with regard to weapon
others have written about, have re- autonomy, would you consider
Scharre ceived less attention. concepts such as “meaningful
I am most concerned about strategic I have perhaps a cynical view human control” or “appropriate
issues. I think that they’re relatively about the legal issues, which is to levels of judgment” or other such
understudied compared to other is- say that I think that countries care concepts currently being developed
sues. There has been a lot of writing about the laws of war about as much helpful in terms of making the use
about the legal issues surrounding as they want to. So you’ll have some of weapon autonomy less risky?
autonomous weapons. The conversa- countries that will pay more atten-
tion today has a focus on laws of war tion to issues surrounding compli- Scharre
compliance and humanitarian con- ance with international humanitarian I’m less concerned about the bump-
cerns in part because of the way the law, and you’ll have countries that re- er sticker. Sometimes people get
issue has been brought to the fore ally don’t, and then you have some in fixated on labels, like meaningful
internationally. That was originally the middle that may be more flexible human control, appropriate human
Christof Heyn’s report as a Special with how they interpret the law but judgment, necessary human involve-
Rapporteur for extrajudicial killings in are going to at least be giving lip ser- ment, or any number of other formu-
the UN and his very influential report vice to it. And I am not sure that more lations. I am less interested in that.
in 2013 that helped raise the issue, in statements and treaties are really I’m more interested in where the dis-
addition to influential and signifi- going to change a country already cussion at the Convention on Certain
cant work by the humanitarian dis- inclined to ignore existing legally Conventional Weapons at the UN in
armament community, for example binding obligations. Geneva has been heading the past
few years, namely trying to get past
the label and talk about what we
think it means.
Imagine robots can do a nything What degree of human involve-
ment should there be in the use of
we wanted, what role should force? That’s valuable because the
technology itself is rapidly evolving.
humans still be playing in Very reasonable people might have
very different assumptions about the
lethal d ecisions – and why? state of the technological maturity
five, ten, fifteen, twenty years from
now. I don’t think anybody really
knows exactly what the future will
look like in terms of the reliability and
robustness of autonomous systems
operating in very complex environ-
ments. So, the appeal of focusing on
the human element is that in princi-
In contrast, I think the strategic ple it should be unchanging!
issues are potentially a greater point The question is not: what can ro-
of purchase between great powers. bots do? It is: Imagine robots can
1 See “The security-policy effects of
There is potentially an area of agree- do anything we wanted, what role
digitisation: Future forms of conflict and
ment worth exploring. My suspicion should humans still be playing in
conflict management”, Metis Study No. 1
( February 2018). is that appealing to humanitarian lethal decisions – and why? I think
8 / 16
Metis Interview | No. 4
Human machine interaction – Paul Scharre
that is an interesting question wor- out how to effectively employ be used to destroy targets that have
thy of exploration and very relevant weapon autonomy? For instance, been chosen by humans. But once
from a legal or ethical standpoint. do you consider the recent Nagorno released many of them cannot be re-
So, from the position of international Karabakh experience a glimpse called – they have an on-board seek-
humanitarian law: what are the min- into the future? Is this the most er, they can sense a target, they have
imum necessary degrees of human likely way we will see autonomy constrained autonomy. You might
involvement in lethal decisions? From unfold on the battlefield? imagine them like an attack dog with
an ethical angle: where do we really blinders on. Those have also been
think it’s vital to have humans in- Scharre around for decades. In fact, the first
volved – are there morally necessary Great questions. Let me take those precision-guided weapons date back
human decisions? in sequence. I think in terms of how to World War II.
Finally, it’s worth looking at human people are employing autonomy, And even prior to that there were
practice in the past. So even in very one of the things that I struggled with various types of automation in weap-
complex systems we might have hu- when I was writing “Army of None” is: ons coming out of the industrial age.
mans physically remote already, and when does this story start? It clearly And so my research brought me all
we might ask how we feel about ele- doesn’t start today, or in 2012, or in the way back to the American Civil
ments of that changing. I would love 2009. I started to try to understand War and to the Gatling gun, where
to see the international community how much autonomy already exists automation accelerated the killing
and various experts explore whether in weapons. potential, the lethality of troops on
there is agreement or whether there Certainly we’ve had automated the battlefield, in ways that did not
are clear differences in points of view, defensive systems in militaries since reach dramatic effectiveness in the
with different camps emerging with the 1980s that in many cases have American Civil War but came to fru-
different perspectives on the future autonomous modes of operation ition during wars of British colonial
of human involvement. Ideas about where a human can turn a switch expansion and then in really violent
“rules of the road” or “codes of con- and the system will automatical- ways in World War I.
duct” surrounding autonomous sys- ly defend a land base or a ship or a Conceptually, there’s a sharp dis-
tems and how they might interact in ground vehicle with the human in a tinction between weapons were
military environments might also be supervisory mode. They can disable humans are making decisions rather
useful confidence building measures the system or maybe halt its oper- than machines. That seems clear in
for states to explore, and so I don’t ation, but otherwise it’s effectively principle. But when you start to look
know that we should put all our eggs operating as an autonomous weap- at the nitty-gritty details of weapons,
in the “human control” or “human on. It’ll engage incoming threats, it gets blurry. I talk in the book about
judgment” basket. Other things aircraft, rockets, missiles, all by it- weapons that are the next evolution
might also separately be worth for self. That’s been around for several of increased autonomy like the Brim-
states to discuss. decades. Then there are precision stone missile or the LRASM that are
guided weapons. They have a much not autonomous weapons – but they
Sauer more narrowly constrained element add more autonomous features.
Let’s put a question mark on of autonomy. But there’s some au- So, I look at Nagorno Karabakh as
weapon autonomy in a different tonomy – many of them are fire-and- one more piece of evidence towards
sense. Has anyone truly figured forget weapons that are intended to the significance of drone warfare
and robotic weapons. We’ve already
seen drones and robotic weapons be
used in other contexts, for example
remote weapon stations and very
Oftentimes things are not crude and unsophisticated ground
robotics in Syria on multiple sides.
as autonomous as they are We’ve certainly seen crude home-
made drones being repurposed.
advertised to be. But I’m also waiting to hear more
details about the degree of autono-
my of some of the loitering systems
used in Nagorno Karabakh. Because
one of the things I found is that
oftentimes things are not as auton-
omous as they are advertised to
9 / 16
Metis Interview | No. 4
Human machine interaction – Paul Scharre
be – which is really interesting. There advanced “wonder weapons” that we’ve already seen in Syria – but
also have been claims made by the are going to drive the divide be- autonomous mass drone attacks at
US military about the degree of au- tween more advanced nations and larger scales being used by non-state
tonomy of Chinese systems, some less advanced nations. Actually, most groups over the next five to ten years.
finger pointing if you will, and I’m a scholars following this closely saw
little bit sceptical. I have less insight the opposite, namely that this tech- Sauer
into Chinese weapons developers, of nology has already widely prolifer- You were mentioning “hype”. Some
course, but my experience with talk- ated to countries around the globe argue that the US, China and Russia
ing to Western weapons developers and into non-state groups. And au- are in an arms race. But maybe
in the US or the UK has been that the tonomy is actually one of the most Missy Cummings is right, and what
actual functionality just doesn’t add accessible components of it! You can we are actually in is a “hype race”.
up to the hype on the website. So, I buy a DJI drone that has more auton- She makes a great case for why
hesitate to make a claim about the omy than an Air Force Reaper drone. many of the military applications
degree of autonomy in the systems The cost of systems scale with size. that are currently envisioned
seen in Nagorno Karabakh. A large robotic aircraft that can be and prototyped will end up not
This also points to what is one used as an intercontinental stealth delivering on what is promised. In
of the early misconceptions about bomber, that is only going to be in that sense, could we end up in a
robotic systems, in part because of the realm of really advanced nation worst-of-both-worlds-situation in
some of the most attention-grab- states. But autonomy itself is fairly ac- which few of the military benefits
bing uses of drones by the United cessible. Hence I suspect that it won’t actually materialize but the risks
States in a counterterrorism role. It is be long before we see not only an in- of political distrust and military
this narrative of drones being these crease in mass drone attacks – which competition become real?
Fig. 4 An X-47B, a technology demonstrator used for testing autonomous operations, conducts a touch and go landing on the flight deck of the aircraft
carrier USS George H.W. Bush (CVN 77). Atlantic Ocean, May 14, 2013. | Source: wikipedia.org, Photo: US Navy photo courtesy of Northrop Grumman by Alan Radecki