"All-in-all, I think it is possible that, thanks to AI research and advanced robotics., we will wage safer, more human-friendly wars in the future."

I don't think you actually understand what a war is supposed to achive or how...

But such a "human friendly smart bot" will not win any wars that we would understand today. Further it would fail to the simple and obvious "Gandi Attack".

That is it would stand on a street corner impotent against a civilian population who just compleatly ignore it or sit down around it pasivly.

That is provided they don't agress it will have no choice but to just sit there doing nothing, until it either fails, malfunctions or is redeployed in a different manner.

One of the problems that people often fail to realise when talking about "humain warfare" is that a war is about applying phisical means (force) for political objectives.

That is a civilian populas or their leaders have to be cowed into a subordinate position and accept the dictat of their opposition.

This requires that they understand that failure to acquiesce has clear, immediate and easily understandable conciquences.

Unfortunatly you also have to realise that a populas that has nothing to lose or does not fear death cannot be easily cowed or defeated and it will rapidly descend into a war of attritian.

Worse if the people or their leaderd do not have the same level of humanitarian feelings as their opponents then the opponents are fighting from what is effectivly a losing position.

A robot restricted by humanitarian ideals stuck on a street corner suronded by passive people is impotent.

It has to be able to carry out effective sanctions or it will fail. Therefor unless programed to inflict sanctions that effect humans in a clear and unambiquous way it cannot be used to prosecute a succesfull war.

Which effectivly means that civilians will get hurt or the robots will be destroyed by them...

Failure to understand the basic realities of war is one of the reasons we are currently in the mess we are in.

Look at the Government of Israel for instance, they have for many years tried all manner of physical coercian to get the palistinians to acquiesce to their dictat.

It has failed and will continue to fail unless they find an effective sanction.

The Palistinians have shown that they are prepared to lose lives and infrestructure on mass and they obviously know that the rest of the world does not have the stomach to alow the Government of Israel to perform ethnic cleansing of the Palistinians.

So unless the government of Israel decides to go down the "ethnic cleansing" route their options are limited to accepting the current stalemate war of attrition or sitting down and negotiating a proper, fair and binding peace.

]]>
2009-04-11T21:36:54Z2009-04-11T21:36:54Ztag:www.schneier.com,2008:/blog//2.2054-comment:364287Comment from UNIT 01 on 2009-04-11UNIT 01
While I agree that reckless development and deployment of autonomous armed devices might pose a threat, I still think that, if properly implemented (yup, I know that is one hell of a caveat) these “fully automated armed military systems” will lead to more precise application of military force and less civilian bloodshed and suffering.

A war waged through properly implemented autonomous “war-bots” will be more humane towards noncombatants.

Why?

Because:
1) A fully automated armed military system is not bound to have self-preservation as strong as a human combatant.
While human soldiers are likely to use lethal force in response to a slight hint of unfriendly behavior (even if ordered not to do so) due to inherent human self-preservation instincts and combat stress, a “war-bot” which lacks such human instincts will strictly follow the “innocent until proven guilty” maxim if ordered to do so, thus preferring to err on the side of misdetecting a combatant over accidentally killing the proverbial “girl who tries to offer war-bot an ice-cream”.

A properly implemented fully automated armed military system will not suffer “morale loss” over the fact that following such orders increases the threat to its existence.

Also, a war-bot following such “civ-friendly” orders will not incur the horrible costs in regards to morale at home, as there are no friends or relatives to mourn a war-bot.

3) Properly implemented war-bots will not develop “war-time nationalism”, thus will not be hostile to all people belonging to the ethnicity common among enemy combatants

4) A properly implemented fully automated armed military system will not demonstrate “personality traits” (not a term that suits a robot well, but I hope you forgive me using it to describe certain behavioral patterns common to humans) that are likely to instill hatred and hostility in civilian populations in the warzone and at home (robots will not throw puppies)

5) A properly implemented fully automated armed military system will not demonstrate negative “personality traits” usually arising due to war-time stress and overall criminologenic environment of war (robots will not rape women “because they can)… Unless programmed to do so ;)

6) A properly implemented fully automated armed military system will demonstrate “personality traits” that will instill trust, friendliness and generic desire to support the “war-bots” in allies and non-combatants.

7) Once a “skill” (like “detect suspect suicide bombers from gait, heartbeat and other behavioral and remotely-detected physiological properties”) is implemented in AI, replicating it to other machines is cheap and easy. Replicating skills in humans is expensive, hard, and not very reliable.

Off course, “civ-friendly” war-bots will suffer additional losses due to guerrillas taking advantage of their “civ-friendliness”, but the pace of technological progress will likely make the costs of each individual autonomous war machine comparable to that of training and equipping a human. Also, the sheer superiority an advanced war-bot will have in terms of armor, firepower and sensor capabilities will make it harder for human enemy to take advantage of.

It is also possible that AI will have additional analytical capability allowing them to deduce hostile intent from evidence that would not be sufficient for a human mind to reach same conclusions (it does not mean the AI will be “smarter than human”, it will be merely “better at spotting IEDs”)

All-in-all, I think it is possible that, thanks to AI research and advanced robotics., we will wage safer, more human-friendly wars in the future.

]]>
2009-04-11T16:31:51Z2009-04-11T16:31:51Ztag:www.schneier.com,2008:/blog//2.2054-comment:363521Comment from Sutobot on 2009-04-08Sutobot
@Anonymous - oh , so you know that War Games were not based on a fictional scenerio.Good !!! It DOES NOT matter on what level robot srews up , because if there were no safety to validate the alarm the misles would go up.Simple , isn't it? We are talking about autonomous robots here.Look up autonomous if you don't know the word's meaning.NO SAFETY , NO VALIDATION - SELF DECISION. And safety block? Give me a break.First wacko is going to disable it and I bet a few will die in labs.]]>
2009-04-08T17:42:00Z2009-04-08T17:42:00Ztag:www.schneier.com,2008:/blog//2.2054-comment:363272Comment from Anonymous on 2009-04-07Anonymous
@Sutobot:
No. You might be thinking of the fictional movie, "WarGames". Or you might be thinking of one of the several incidents where alert systems caused preparations for launch to begin. However, real nuclear silo control procedures are not at all autonomous and have several layers of human checks and balances. Contrary to some sensationalistic accounts, none of those incidents actually got beyond initial preparations, because they incorporated backup checks which failed to validate the initial alarm. ]]>
2009-04-07T21:03:52Z2009-04-07T21:03:52Ztag:www.schneier.com,2008:/blog//2.2054-comment:363265Comment from Sutobot on 2009-04-07Sutobot
Didn't we have an autonomous robotic systems in nuclear silos and it almost caused a nuclear war in real life???What year was it???]]>
2009-04-07T20:31:06Z2009-04-07T20:31:06Ztag:www.schneier.com,2008:/blog//2.2054-comment:363264Comment from Autobot on 2009-04-07Autobot
It's simple.Whatever we ever design or develop will have a mirrir reflection in a "dark side".WHY do we habe to make prophets of all classical Sci-Fi writers:Assimov, Stanislaw Lem , Arthur C.Clark ,Phillip K.Dick? They wrote about that stuff decades before.Do you still consider Bladerunner or Terminator a sci-fi? If you do, than you are a naive person.How about "War Games" or the awesome "I Robot".Can we ever learn that every great discovery brings destruction and THAT is what needs to be controlled.Machine can never make a correct decision , because most of our life is built on what most robots(morons) would consider not logical at all.Not all logical moves result in logical outcome.I love a comment with chess....]]>
2009-04-07T20:28:27Z2009-04-07T20:28:27Ztag:www.schneier.com,2008:/blog//2.2054-comment:331124Comment from ChrisTheEngineer on 2008-12-17ChrisTheEngineer
There is no possible way that for the foreseeable future that "ethics" can be coded. None. ]]>
2008-12-18T01:36:55Z2008-12-18T01:36:55Ztag:www.schneier.com,2008:/blog//2.2054-comment:330425Comment from The other Alan on 2008-12-15The other Alan
My question is: why do they need to be autonomous? We have unmanned (yet, human controlled) armed drone aircraft, why the need to make these autonomous? Anyone?

I mean, I can understand the desire to want to do so, but the downside is just to huge to ignore.

]]>
2008-12-15T18:55:36Z2008-12-15T18:55:36Ztag:www.schneier.com,2008:/blog//2.2054-comment:330402Comment from bob on 2008-12-15bob
Considering how many times in a single day I am thwarted from a reasonable goal by the invalid assumptions of the person who programmed the logic of the device in question (Chevrolet windshield wipers & vent controls, Windows Vista, elevators just to name a couple) I am scared by the idea of an 'autonomous' device capable of killing people (I mean as its end design goal, not by accident - machines have probably been killing people by accident since right after the wheel was invented).]]>
2008-12-15T16:21:28Z2008-12-15T16:21:28Ztag:www.schneier.com,2008:/blog//2.2054-comment:303097Comment from Richard Garlow on 2008-09-01Richard Garlowhttp://www.nothostedhere.com/technology/weapons-of-war/46-fully-armed-robots-on-patrol-in-iraq
Check out the killer robots they deployed in Iraq..

]]>
2008-09-01T22:41:48Z2008-09-01T22:41:48Ztag:www.schneier.com,2008:/blog//2.2054-comment:281921Comment from drachenchen on 2008-06-26drachenchen
Looks like I missed the discussion, along with most of the American people. This is typical of most weapons programs. The very idea that with the cult of secrecy, AND the cult of brutality, in place, that anyone connected with the military is qualified to develop an "ethical weapons system" of any kind, is beyond laughable. It is perhaps laughably obscene, in the vilest sense. This nation has just finished invading a foreign nation on false pretenses for the proven purpose of stealing their natural resources. This nation's military has killed over HALF a MILLION "non-friendlies", and maimed many more, including children, while officially "keeping civilian casualties to an absolute minimum" with "tightly targeted munitions".

Calling it a "smart bomb" obviously doesn't make it smart. When retarded whackos are calling the targets, that makes it a "retarded whacko bomb". When fascio-Christian nut-bars are telling it where to land, it becomes a "fascio-Christian nut-bar bomb". Insert either of those two modifiers in place of "ethical" in your discussions of autonomous battlefield robots, and you'll get a far more accurate picture of how they will behave. "Yes, we're building an autonomous crypto-capitalist, greed-oriented battlefield robot. It will only kill people standing in the way of taking OUR oil, or OUR food away from them."

-And now that the Bushies have designated North America as an operational theater for the very first time, think about how it will be to have one of these killing machines on every street corner...

I also find it hilarious how there is NO discussion about how you will manage to prevent damage to the robot's so-called "ethical" circuits. You know, damage? -Like what happens during actual battle?

Oh, and one last thing, how will you folks who are even THINKING about developing these things find out the meaning of the word "ethics"?

]]>
2008-06-26T11:59:28Z2008-06-26T11:59:28Ztag:www.schneier.com,2008:/blog//2.2054-comment:250990Comment from aikimark on 2008-02-27aikimark
Prof Noel Sharkey to address this at University of Sheffield today:http://www.physorg.com/news123306321.html
]]>
2008-02-27T15:47:21Z2008-02-27T15:47:21Ztag:www.schneier.com,2008:/blog//2.2054-comment:241652Comment from kmax on 2008-01-30kmax
In contrast to most people's comments, I think such an ethical
robot could be a very good thing, even considering myself a pacifist.

As history has shown, war is inevitable. Most unfortunate.
As history has shown, soldiers commmit crimes and kill innocent people.
Most unfortunate, too.
As history has shown, software has bugs, so will the robots,
and kill innocent people. Most unfortunate, once more.

The question is: who is more likely to follow ethical rules and the
Geneva convention in extreme situations?
Who will end up with the smaller number of killed innocents,
a group of n soldiers or n robots?

]]>
2008-01-30T16:00:37Z2008-01-30T16:00:37Ztag:www.schneier.com,2008:/blog//2.2054-comment:241459Comment from Diodotus on 2008-01-29Diodotushttp://www.elected-swineherd.blogspot.com
A great deal of faith is being placed here on the idea that the generals, civilian policymakers, and their minions in the R&D industries want the troops to behave well, and it's just the bad apples who muck things up. A lot of history suggests otherwise. Maybe we need robot robot-programmers, as well...
]]>
2008-01-30T05:58:31Z2008-01-30T05:58:31Ztag:www.schneier.com,2008:/blog//2.2054-comment:241285Comment from Rob on 2008-01-29Rob
Things don't always go according to plan when you allow a device with limited capacity for reasoning onto the battlefield.

]]>
2008-01-29T23:53:54Z2008-01-29T23:53:54Ztag:www.schneier.com,2008:/blog//2.2054-comment:241182Comment from DaveK on 2008-01-29DaveK
The whole thing seems like a massive exercise in question-begging to me.

"If only we knew how to make an autonomous battlefield robot, we could make an autonomous ethical battlefield robot, if only we knew how to make it ethical".

Well, think of the two-man rule for launching nuclear weapons as part of the fail-safe, but, yeah...

"I built the M-5 with my engrams" - Dick Cheney.

If they built the M-5 with Dubya's engrams it wouldn't be able to add, much less do any real damage.

]]>
2008-01-29T20:10:44Z2008-01-29T20:10:44Ztag:www.schneier.com,2008:/blog//2.2054-comment:241133Comment from badfrog on 2008-01-29badfrog
These implications were all worked out in science fiction in the fifties and sixties. The first (I think) was "Cordwainer Smith's" manshonyaggers (menschenjaggers, or manhunters), which were still patrolling their area after 50,000 years. Probably the most thorough were the Berserker stories of Fred Saberhagen. I remember the BOLOs, too. Norman Spinrad wrote the Star Trek story "The Doomsday Machine," which introduced the idea to a wider audience without any real ethical implications. There must be hundreds of stories laying about in moldering stacks of old Galaxy, Analog, and Amazing magazines.

It takes twenty years to raise a soldier, and months to train him well.

If you can manufacture and program them by the millions, it's much cheaper in terms of time, emotion, and supply. Attrition is now an attractive option. Think of fighting the Iranian or Chinese "human wave" strategy with a "robot wave."

However, the U.S. combined arms strategy is far more effective in terms of massed battle. Bombs are much cheaper than robots

But think of sending a robot with advanced AI and a machine gun into a crack house in Detroit or a safe house in Baghdad. The thinking is to save some good guy lives.

But it will be just as safe and still more effective to send in a robot controlled by a human hand and mind, such as the flying drones currently employed over Iraq, Afghanistan, and Israel.

For the forseeable future, robot warfare will probably be defensive, i.e., a machine gun that fires on anything that crosses an infra red beam. Kind of like an automated grocery store entrance, only noisier.

]]>
2008-01-29T13:45:29Z2008-01-29T13:45:29Ztag:www.schneier.com,2008:/blog//2.2054-comment:240911Comment from Malcolm on 2008-01-29Malcolm
Bruce, you missed another "daunting problem": doubt.

Even if we nail the problems you list, we still end up with a bunch of killing machines that are absolutely certain that they're doing the right thing. History shows that that's a recipe for genocide.

]]>
2008-01-29T11:14:31Z2008-01-29T11:14:31Ztag:www.schneier.com,2008:/blog//2.2054-comment:240847Comment from averros on 2008-01-29averros
Oh, how cute. The people manifestly lacking any ethics (in that they think nothing of wasting money expropriated from people on advising how to do automated mass-murder "ethically") are writing about robot ethics.

There's nothing complex about the issue of killer robots. They're simply elaborate traps - and all "ethics" of such traps comes from the intent of their use by their owners.

]]>
2008-01-29T09:09:31Z2008-01-29T09:09:31Ztag:www.schneier.com,2008:/blog//2.2054-comment:240835Comment from SolarSauna on 2008-01-29SolarSauna
It is not clear if this type of ethical control and reasoning system for robots could prevent incidents like: a) the genocide in Rwanda, b) ethnic cleansing in Kosovo, c) forcing children to fight in armies like in Liberia and Congo, d) cutting off a hand of the enemy to overload their medical facilities as in Liberia. Once these robotic war fighters are developed they would become the prime objects for sale by unethical(?) arms dealers. What's to prevent their algorithms from being altered to gain an advantage in combat. Will the software designers who made a mistake or an unauthorized alteration be charged with a war crime?]]>
2008-01-29T08:07:22Z2008-01-29T08:07:22Ztag:www.schneier.com,2008:/blog//2.2054-comment:240752Comment from Jim Lux on 2008-01-28Jim Lux
Didn't Norbert Wiener (of "Cybernetics" fame) discuss this (the ethics/morals of automated systems that kill people) in some of his works 50 or more years ago? I can't think of the titles off hand. "God and Golem" or "The human use of human beings" might be the ones.

It comes down to a very complex issue of where do you draw the lines. How is sending targeting information to a B52 (with a human pilot) flying above the clouds different than sending the same targeting info to a ICBM?

]]>
2008-01-29T02:37:14Z2008-01-29T02:37:14Ztag:www.schneier.com,2008:/blog//2.2054-comment:240726Comment from paul on 2008-01-28paul
This kind of work also reminds me a bit of the work Mark Stefik at PARC did in the 90s on rights definition languages for digital content -- machine-readable, formal languages that would allow content publishes and content purchasers to negotiate how much the purchasers would pay for how many copies made under what conditions and so forth. It was very elegant, but since the only position most publishers wanted was "We keep everything. Take it or leave it." the idea never reached terribly widespread distribution.]]>
2008-01-29T01:12:57Z2008-01-29T01:12:57Ztag:www.schneier.com,2008:/blog//2.2054-comment:240717Comment from Bond on 2008-01-28Bond
We're trying for the wrong standard. Trying to make 'ethical autonomous weapons' is not the issue, trying to make 'more-ethical-than-humans autonomous weapons' is undeniably an excellent idea. While it may well be very difficult to make truly ethical robots, the standard of making robots that behave themselves better than humans is pretty easy, really.
It should be noted that armies go to considerable efforts to prevent their soldiers making decisions, hence 'rules of engagement' rather than an instruction to 'be ethical'.
We worry about robots turning against us and destroying us, but we should remember that humans have been turning against each other and destroying each other for a long time, and it's still easier to make more people than it is to make more robots. When robots have a mechanism in place to evolve, then we're in trouble.]]>
2008-01-29T00:30:22Z2008-01-29T00:30:22Ztag:www.schneier.com,2008:/blog//2.2054-comment:240716Comment from ZaD MoFo on 2008-01-28ZaD MoFo
The robots will become an extension of our actions, both in our factories and in our armed gestures. Is it desirable to choose a model of human cognition to draw up behaviour (with its invariable weaknesses and the ability to use social engineering to neutralize them) to be this model? The era of Terminators seems close if one considers that modern homes will soon present several robot like the VCR and it was less than twenty years ago.]]>
2008-01-29T00:25:57Z2008-01-29T00:25:57Ztag:www.schneier.com,2008:/blog//2.2054-comment:240715Comment from NotME on 2008-01-28NotME
We don't need fancy ST plots with simulated war, or even fanciful EMP devices like The Matrix.

All combatants just need to wear CAPTCHAs to confound the machines for days.

@ all those who called this reprehensible
Amen! War is bloody and -usually- profitable. Without the blood, it's just profitable. War without blood = murder for hire.

]]>
2008-01-29T00:19:07Z2008-01-29T00:19:07Ztag:www.schneier.com,2008:/blog//2.2054-comment:240707Comment from dio gratia on 2008-01-28dio gratia
Politicially correct autonomous killing machines? Shirley, you jest...]]>
2008-01-28T23:29:54Z2008-01-28T23:29:54Ztag:www.schneier.com,2008:/blog//2.2054-comment:240702Comment from Peter E Retep on 2008-01-28Peter E Retep
As a think piece, it's barely O.K.
As a command rubric, the bias
in favor of presuming total obedience of a system is a serious - perhaps literally fatal - flaw.]]>
2008-01-28T23:04:29Z2008-01-28T23:04:29Ztag:www.schneier.com,2008:/blog//2.2054-comment:240697Comment from nu on 2008-01-28nu
This country is already wiping its ass on the Geneva Conventions.]]>
2008-01-28T22:36:42Z2008-01-28T22:36:42Ztag:www.schneier.com,2008:/blog//2.2054-comment:240686Comment from woot on 2008-01-28woot
I look forward to these rules as an addition to Genevia Convention rules - (or a new version of them). Understanding how and when robots should be used in warfare is something we should do.

I don't think it would be too hard for one disciplined in the arts to take an RC car, add an imaging sensor (for face/place recognition), and program it to carry a "package". I would prefer there be rules about how close the next non-target person is for a device like that. (Of course - I would prefer this thing to not exist at all...)

Granted - not all people would agree or use these rules. But the more countries that do, the safer each of us is going to a baseball game.

]]>
2008-01-28T22:18:52Z2008-01-28T22:18:52Ztag:www.schneier.com,2008:/blog//2.2054-comment:240662Comment from Jack C Lipton on 2008-01-28Jack C Lipton
The complaint about dredging up Sci-Fi scenarios is that Sci-Fi and other speculative fiction tends to be better at illuminating dystopias rather than utopias... because dystopias tend to be what we NEED to worry about.

Beyond that, looking at what passes for political leadership we have, who set the ethical rules? Who reviews them?

Personally, I want a living thing somewhere in the decision loop.

]]>
2008-01-28T21:26:52Z2008-01-28T21:26:52Ztag:www.schneier.com,2008:/blog//2.2054-comment:240653Comment from fusion on 2008-01-28fusion
Then there is the fundamental problem sketched by Korzybski; semantics. Excuse my fumbling to bring this to the surface. Arkin may be wise; but words are abstractions linked to real-world processes by intellectual structures - which humans construct and use differently.

Whose "ethical" are we using?

]]>
2008-01-28T20:53:26Z2008-01-28T20:53:26Ztag:www.schneier.com,2008:/blog//2.2054-comment:240640Comment from Roy on 2008-01-28Roy
The slippery slope began with self-propelled torpedoes and naval mines. Land mines followed soon enough.

Now we will see autonomous weapons actively discriminating among friendlies, hostiles, and neutrals. In the 'weapons tight' condition the rule is to kill only hostiles; in 'weapons free', it will kill anyone not friendly.

What exactly does a hostile look like? A friendly? A neutral What algorithm will tell them apart?

If we equip friendlies with transponders, then most of the time the automaton won't kill them, but it will kill all the non-transponders it can -- hostiles, neutrals, and unlucky friendlies. Building better transponders might seem like a good idea, but the money is made mass-producing them, so once mass-production begins, mass-counterfeiting begins.

Killbots would be safer than indiscriminate weapons like a hydrogen bomb, but why would we want a weapon that will kill our own kind, and will kill neutrals indiscriminately? I thought the whole point of war was to make fortunes building weaponry, and then make bigger fortunes replacing the weaponry destroyed or outmoded.

]]>
2008-01-28T20:31:27Z2008-01-28T20:31:27Ztag:www.schneier.com,2008:/blog//2.2054-comment:240629Comment from Charles Choi on 2008-01-28Charles Choi
Kyle has nailed this, especially when looked in the light of asymmetric warfare; using robots to do the dirty work of war will only enhance the morale of those being shot at. Think of it from the other side: "they're so cowardly that they have to send machines to kill us."]]>
2008-01-28T20:10:36Z2008-01-28T20:10:36Ztag:www.schneier.com,2008:/blog//2.2054-comment:240625Comment from havvok on 2008-01-28havvok
"The creation of techniques to permit the adaptation of an ethical constraint set and underlying behavioral control parameters that will ensure moral performance, should those norms be violated in any way, involving reflective and affective processing."

That is far and away the scariest thing I have ever read in the context of weapons development.

The ability to detect and select new targets effectively on the battlefield would be very cool, the ability to analyze and infer new rules about target selection, based on machine interpreted rules about morality, not so much.

]]>
2008-01-28T19:43:18Z2008-01-28T19:43:18Ztag:www.schneier.com,2008:/blog//2.2054-comment:240623Comment from Kyle Reese on 2008-01-28Kyle Reese
When machines fight against humans, humans are always the good guys]]>
2008-01-28T19:39:49Z2008-01-28T19:39:49Ztag:www.schneier.com,2008:/blog//2.2054-comment:240618Comment from old guy on 2008-01-28old guy
It all sounds pretty scary, and really, really sad.]]>
2008-01-28T19:33:44Z2008-01-28T19:33:44Ztag:www.schneier.com,2008:/blog//2.2054-comment:240615Comment from Ross Dmochowski on 2008-01-28Ross Dmochowskihttp://www.samsungtechwin.com/product/features/dep/SSsystem_e/SSsystem.html

All Your Base Are Belong To Us!

]]>
2008-01-28T19:25:12Z2008-01-28T19:25:12Ztag:www.schneier.com,2008:/blog//2.2054-comment:240614Comment from am on 2008-01-28am
If we are concerned about a future desensitized to war, we're already there. I feel so far removed from the daily horrors of a war my country is currently fighting that it is almost like it is already being fought with robots rather than humans. ]]>
2008-01-28T19:20:36Z2008-01-28T19:20:36Ztag:www.schneier.com,2008:/blog//2.2054-comment:240611Comment from pavel on 2008-01-28pavel
When my company made plans to work in the defense and security sector, I was tasked to come up with a set of rules defining how far we would go there.

For me it is generally immoral to create or build a machine that automatically kills people. The mentioned Land Mines also fall into this category, as well as systems like the Oerlikon GDF-005 robot cannon that killed nine soldiers in south africa in October 2007.

When I build a gun and someone pulls the trigger, he has to cope with it.

When I create a rule based system that pulls the trigger, it is actually me who makes the final decision who gets killed.

Even if I could create a perfect and flawless system, it still might kill anyone under certain circumstances, and that includes myself or my kids.

However, in reality every such system will be heavily flawed, and it *will* kill innocent people sooner or later, even if it is a simple system like a land mine.

Using a complex AI-based system today or even in the foreseeable future for the purpose of making kill decisions is stupid and dangerous bullshit, given that we did not make much real progress in the last 60 years of AI.

I think *any* machine with the purpose of harming people that does not require a conscious and present human decision to do so should be outlawed everywhere and forever, no exceptions.

Autonomous killing machines are not only a very bad idea, they are highly condemnable.

No amount of built-in ethics can change that.

]]>
2008-01-28T19:07:41Z2008-01-28T19:07:41Ztag:www.schneier.com,2008:/blog//2.2054-comment:240597Comment from Dom De Vitto on 2008-01-28Dom De Vittohttp://devitto.com
The long-standing Sci-Fi prophecy of intelligent machines rising up to enslave and destroy the human race has disappeared from modern culture.

As far as I can tell, this coincided with the release of MS-DOS.

]]>
2008-01-28T18:42:46Z2008-01-28T18:42:46Ztag:www.schneier.com,2008:/blog//2.2054-comment:240596Comment from peri on 2008-01-28peri
For those who wouldn't recognize Ronald Arkin's name, he literally wrote the book on modern robotics; see "Behavior-Based Robotics."

The article does not mention Terminator style movies but he does directly address Asimov's laws:
----------------------------------------------------------------

I suppose a discussion of the ethical behavior of robots would be incomplete without some reference to [Asimov 50]’s "Three Laws of Robotics" (there are actually four [Asimov 85]). Needless to say, I am not alone in my belief that, while they are elegant in their simplicity and have served a useful fictional purpose by bringing to light a whole range of issues surrounding robot ethics and rights, they are at best a strawman to bootstrap the ethical debate and as such serve no useful practical purpose beyond their fictional roots. [AndersonS 07], from a philosophical perspective, similarly rejects them, arguing: "Asimov’s ‘Three Laws of Robotics’ are an unsatisfactory basis for Machine Ethics, regardless of the status of the machine". With all due respect, I must concur.

]]>
2008-01-28T18:41:59Z2008-01-28T18:41:59Ztag:www.schneier.com,2008:/blog//2.2054-comment:240590Comment from Seth Gordon on 2008-01-28Seth Gordonhttp://dynamic.ropine.com/yesh/
If an autonomous military robot violates the Geneva Conventions, would its programmer be liable for war crimes?]]>
2008-01-28T18:33:25Z2008-01-28T18:33:25Ztag:www.schneier.com,2008:/blog//2.2054-comment:240583Comment from Michael Richardson on 2008-01-28Michael Richardsonhttp://www.sandelman.ca/mcr/
@possibly those who voted for the party in power.

I'm trying to figure out how that would work in Florida. Do they have to *know* that they voted for the party in power? (i.e. does it need to know their intent?)
And if they intended to vote for the party in power, but failed due to hanging chad, are they spared?

]]>
2008-01-28T18:07:01Z2008-01-28T18:07:01Ztag:www.schneier.com,2008:/blog//2.2054-comment:240580Comment from paul on 2008-01-28paul
If you tell a robot tank to shoot a group of prisoners, does that absolve you of responsibility for a war crime?

From the short version, it sounds as if this may be a much more sophisticated version of the limits coded into CIWS and other self-targeted weapons to stop them from shooting into or through the installation they're supposed to defend.

]]>
2008-01-28T18:02:38Z2008-01-28T18:02:38Ztag:www.schneier.com,2008:/blog//2.2054-comment:240573Comment from Kuroneko on 2008-01-28Kuroneko
Tachikomas for the win, I say]]>
2008-01-28T17:43:38Z2008-01-28T17:43:38Ztag:www.schneier.com,2008:/blog//2.2054-comment:240571Comment from aikimark on 2008-01-28aikimark
I miss the old BattleBots show on Comedy Central. No humans were hurt in the production of that show. Why can't we just fight by proxy? Best engineering wins.

Is it beyond hope that battle mechs might be controled by semi-benevolent AI, like Tweedledee and Tweedledum?

]]>
2008-01-28T17:32:59Z2008-01-28T17:32:59Ztag:www.schneier.com,2008:/blog//2.2054-comment:240570Comment from Sean on 2008-01-28Sean
Didn't we already try this with mustard gas ? The friendly fire will be pretty awesome when it happens. The friendly robot detects soldiers and eliminates them. Can't think of anything fairer than that.]]>
2008-01-28T17:30:37Z2008-01-28T17:30:37Z