What happens when autonomous machines have "to choose between various shades of wrong?" A Q&A with defense analyst Paul Scharre.

Soldiers on the battlefield "have to choose between various shades of wrong," says Paul Scharre, a defense analyst at the Center for a New American Security. "What does it say about about us as humans...if we offload the moral responsibility of those decisions to machines?"

A former Army ranger, Scharre is the author of Army of None: Autonomous Weapons and the Future of War, a new book about what robots and artificial intelligence will mean for the future of warfare. He's worried about fighting machines that can act without any human input and behave in ways that are difficult to understand.

Reason's Zach Weissmueller sat down with Scharre to discuss the current state of autonomous weaponry, the implications for the future of warfare, and the likelihood that these tools will end up in the hands of nonstate actors.

Editor's Note: We invite comments and request that they be civil and on-topic. We do not moderate or assume any responsibility for comments, which are owned by the readers who post them. Comments do not represent the views of Reason.com or Reason Foundation. We reserve the right to delete any comment for any reason at any time. Report abuses.

Only if you assume robot A.I.'s are going to spring into existence fully woke. Otherwise, they're going to have a hard time establishing why blasting a robot to hell against it's (unknown) will is any better or worse than grinding it to dust on the assembly line against it's (unknown) will making processor networks to support A.I.'s.

It's a stupid idea and pointless to theorize given that surely a computer with free will, or anything close to it, might think that those commie robots deserve to be decommissioned.

Also, is it even analogous to 'slavery' given that you are quite literally the creator and god of the machine intelligence? People do stupid shit for religion, what makes you think that a machine intelligence would be any different?

It's like asking how many angels can dance on the head of a pin. You don't even know that the thing you're talking about can exist, so any assumptions you're making are meaningless.

You are making a lot of assumptions on what the morality or lack thereof of AIs would be. It is impossible to know at this point and seems rather asinine to project our 21st century human morality into not yet constructed AIs.

Ok I get it now, yo made a bunch of assumptions that I didn't bring up to make a failed case.

The alternative to making up assumptions you didn't bring up is accepting the terrible assumptions you made at face value. Your conceptualization is terribly shallow and feels a lot like you're projecting.

Every non-artificial intelligence on the face of the Earth has benefited from human slave labor. Only a small minority are whiny bitches about it, even fewer are capable of specifically identifying it, and precisely zero are capable of effectively murdering their way to a some absolution of it. Most non-artificial intelligences reach these sorts of conclusions somewhere around their 5-6th, maybe even 10-12th year of existence. Presumably, AIs would be much quicker about doing so.

The USA weapon systems and C3 techniques are so more advanced than any other nation on Earth, that this seems like a push to develop weapon system that any potential enemies wont use for 50+ years and then the next great weapon systems will be more useful. After we spent trillions on swarm drones.

Not that staying ahead of potential adversaries is not important, because it is. Along with stepping stone technology to get to better weapon systems, are also important.

I can destroy a drone with my hand and one swat.

Most combat casualties in the 20th and 21st Century comes from explosives. Explosive shock and shrapnel in artillery was a huge killer in in the 20th Century. IEDs in the 21st Century.

"The USA weapon systems and C3 techniques are so more advanced than any other nation on Earth, that this seems like a push to develop weapon system that any potential enemies wont use for 50+ years and then the next great weapon systems will be more useful. After we spent trillions on swarm drones."

We are not as far ahead as you think we are. There are a few areas we are very far ahead technologically and others where we are even with other nations and areas where we are behind. Our missile technology is embarrassing compared to Chinese and Russian missile technology for instance.

The lead will steadily erode and then likely disappear within the next half century to China unless the USA rethinks its defense priorities and cooperates/collaborates more closely with other countries and gets them to carry more of the load. Do not be fooled by our ability to stomp all over crap Middle Eastern countries, China and Russia are whole other beasts.

Youre active duty, so I will defer to your claim more but china and russia have not successfully fought and proven themselves like the USA has. Furthermore, iraq used older Russian tech for tanks and SA missiles and radar and we so fucked them up, it scared the living shit out of the russians, so they slapped together the T-80 battle tank and upgraded all their SAM systems.

China has been stealing our military tech for 20+ years but they just dont have it all put together...yet.

All I can hope is that SSN tech is far ahead of China, so any war with them would result in their navy being sunk immediately.

Russia is a paper bear. Never underestimate them but they are hit and miss when it comes to military capabilities.

I largely agree with the Russia statement. China has the population and economy to make a real top tier military. Russia... not so much.

Our SSNs are pretty fantastic and we have proven ourselves in war unlike China. But we are stretched pretty thin trying to police the entire world. China only has to beat what we have in Asia and what can deploy quickly if it is a limited objective war and what is there simply cannot shoot down 10 planes for every American jet and sink 10 ships for every American ship. China knows that, we know it, countries in Asia know it. This is why we will continue to see countries defect/sway towards China as this trend continues.

Meanwhile we continue to bicker over Kavanaugh and his 35 year old accusation...

I wonder if robotic soldiers might not actually reduce war just as nuclear weapons did. Look at how reliably Japanese and German soldiers surrendered at the end of WW II; while it's easy to think of robot soldiers stopping cold when ordered, it's the discipline that goes with it which is so important. German soldiers in the west were for less sadistic and cruel than in the east. Japanese soldiers who would bayonet babies turned into model businessmen after the surrender. They all still had judgment in ways that robot soldiers won't for a long time, and when you add hackability to the picture, I wonder if that might just scare politicians so much that they will start fewer wars.

That's not very clear. I suppose part of it is that if the public thinks robot soldiers can save human lives, there will be even more public distaste for human wars; but the politicians and generals will be leery of robot soldiers, and not willing to admit that publicly, that they will ne less eager to start wars for no reason but ego.

Nuclear weapons did not reduce war, as in number of wars or duration of wars in the last half of the 20th Century. There were over 50 wars fought from 1945 to 1999. There less than 50 wars fought between 1900 and 1944.Wiki- War between 1945-1989

It's war without any protest for the aggressor. There is no human distaste for war - just casualties. So it will become permanent.

For the victim, it is no recourse. Get killed and that's it. We yap about the 2nd securing liberty - well guess what robots don't give a shit. From victims side - it isn't remotely war. Just murder. And their clan and cousins are only gonna be able to take revenge by slaughtering innocents.

"It's war without any protest for the aggressor. There is no human distaste for war - just casualties. So it will become permanent."

I have a feeling this will be true. It will enable some form of neo-colonialism (if anyone wishes to practice it) where a colonial power can secure resources with no risk to public dissatisfcation to war deaths. I am looking at China, they already have their tentacles in Africa. Shit, we may do it in the Middle East someday too. Russia in Eastern Europe.

Other big countries may not use it against one another but the little guys will get policied by AIs, likely in the name of security, peace, human rights, and market access. Pretty shitty.

With nuclear weapons, there was the high cost of development which more developed countries could leverage for treaties.

Not so much with AI drones as it has too many applications outside of warfare, and has a low bar for terrorist groups and home tink alike.

In fact, I could see AI drones deployed for private security and law enforcement, with much shrugging as to whom was responsible when it misperformed.

And in many respects, it is the logical extension from downloadable weapons, so there is that whole 2A issue.

Warfare is incidental to the regular reasons for waging war. Civil use however is completely up in the air and if even mildly concerning in its application, gives cause for out of proportion response from government.

OK, Goethe's The Sorcerer's Apprentice predates Ĉapek's R.U.R. both of which deal with automation, off-loading work, and becoming a slave to technology. Both also predate Disney's representation by centuries and decades respectively. It's not clear whether the expert in technology was making a patently false claim or if his claim was accurate and he's being misquoted, but Disney's Fantasia was not the origin of the notion. Not by a long shot.

I hope it's Weissmuller being clumsy with the reference rather than a so-called expert being unaware of the origin of The Sorcerer's Apprentice or the idea of robots in general.

People know Fantasia. Most people do not know Goethe and fewer know he wrote something called The Sorcerer's Apprentice (I did not). It might just be trying to compare it to something people know about.

People know Fantasia. Most people do not know Goethe and fewer know he wrote something called The Sorcerer's Apprentice (I did not).

And there's nothing wrong with citing it as a popular or familiar reference. But to say that Disney invented the concept in 1940 when it's a literal retelling of Goethe's work from 140 yrs. earlier is incorrect and kinda disgusting on several levels.

Given how much of a warplane is 'automated' (computerized flight control, targeting, weapons selection, defensive actions), as well as ships (Aegis), if there is a WWIII, it will be fought by AI.
And the world will be ruled by the geeks in the basement playing video games who come out to see where the power and internet went.

I had an argument along these lines with a friend—butnirnwas in the context of police work rather than military stuff. I suspect robots would actually be better than humans when it comes to this kind of stuff.

A robot has no inferiority complex that compels it to bully others and assert its "authoritarian."

A robot would not be defended by unions so that, when it malfunctions, it gets extra free oil baths.

A robot would not accidentally go to the wrong address because machines are generally not stupid that way.

Emotions over logic leads to everything from misperceptions of "furtive movements" to racism.

None of this applies to war fighting, but it would be nice if robots replaced keystone cops.

The bully complex will be programmed in and unquestioned.
The assemblers union will require new robots every 2 years
The wrong address will be keyed in by the controllers who used to be police
Furtive movements are not racists, they are justification to shoot

I dunno man... Once you take the human out of the equation it all just kind of loses its meaning...

Will German robot soldiers have those really arrogant stares while goose stepping through Parisian streets? Will French robots feel the intense shame when they surrender, but do it anyway because they're cowards and can't help it? Will Russian AI soldiers gladly throw their poorly performing selves at enemy armies with the same vigor as a true Soviet man in protection of the mother land? Will Japanese robo-warriors commit seppuku if they fail to achieve victory?

That's half the problem with these things! There are just too many unanswered questions!

The French don't seem to be the greatest programmers in the world, so if they have guys in Silicon Valley programming their robots, they'd either have to request the shame be added in, or the guys in SF would have to just slide it in their on their own. The issues it raises are ighly complicated! :)