The Campaign hopes that this briefing will be reflected in the Board’s recommendations to the UN Secretary General, and so additional consideration and engagement on this theme by UN officers and agencies.

***

Presentation to the UN Secretary-General’s Advisory Board on Disarmament Matters

I am speaking on behalf of the Campaign to Stop Killer Robots, an international coalition of non-governmental organizations launched in April this year. My organization, Article 36, is a co-founder of the campaign and a member of its Steering Committee.

We would like to thank you for the opportunity to brief the Secretary-General’s Advisory Board on Disarmament Matters on fully autonomous weapon systems and discuss with you some of the concerns raised by this emerging weapons technology.

The campaign was launched to raise awareness about the implications of fully autonomous weapons (‘killer robots’), and calls for a comprehensive, pre-emptive ban on fully autonomous weapons.

Fully autonomous weapons would be able to fire at targets that they not only detect themselves, but also select on their own, without human intervention being necessary to carry out the attack. The key element is that the machine has the power to ‘choose’ a target of attack independently. Unlike existing unmanned, remote-controlled weapons systems, such as drones, fully autonomous weapons would launch attacks without the involvement of a human operator.

This would pose a fundamental challenge to the legal framework governing the use of force and the protection of civilians and other victims of war, and to the maintenance of peace and security.

Fully autonomous weapons have, to our knowledge, not yet been deployed, but we are concerned that drones, as well as certain unmanned ground or sea-based vehicles, could potentially be given the capacity to make decisions for themselves as technology develops. Some weapons systems that are currently in use already have the capability to attack in fully autonomous mode, though, so far, States have chosen to keep a person involved in, or at least, on the targeting loop. But military and policy documents of a number of States indicate a clear trend towards increasing autonomy of weapons systems, with large amounts of money being allocated to research and development of this capability.

Proponents of increasing autonomy of weapons systems point to military and other advantages that such weapons would have over humans. Yet, as the UN Special Rapporteur on extrajudicial, summary or arbitrary executions, Christopher Heyns, made clear in his report presented to the Human Rights Council last month, fully autonomous weapons raise multiple moral, legal, policy, technical, and other concerns.[1]

The campaign has welcomed the Special Rapporteur’s report and called for States to endorse and implement its recommendations.

One important issue raised by the Special Rapporteur is the concern that fully autonomous weapons would not possess the qualities necessary to comply with international legal rules devised to protect civilians, wounded soldiers, and those who have surrendered, in times of armed conflict.

Human agency and judgment, appreciation of the context, understanding of the intentions behind people’s actions, and anticipation of the direction in which events are unfolding are necessary for the application of basic rules of international humanitarian law. We are concerned that fully autonomous weapons would lack situational awareness and morality, and that they could not mimic human decision-making processes in a way that would enable them to evaluate the unpredictable circumstances that arise in most operational environments.

In addition, whilst removing human soldiers from the battlefield can protect those soldiers’ lives, it would also further shift the burden of armed conflict onto civilians, and accentuate the asymmetry in confrontations between those with high-tech weaponry and those who do not possess such weapons – features of modern conflict that are already seen to be highly problematic from a humanitarian perspective.

There is also great concern that the use of fully autonomous weapons systems would create an accountability gap. It may be difficult to establish who is responsible for the harm caused in an attack involving a fully autonomous weapons system. Even if a responsible party can be identified, there is no clarity on who would be legally accountable for a robot’s actions: the commander, the programmer, or the manufacturer? Unlike humans, fully autonomous weapons cannot take the blame for wrongful acts.

We are very concerned that without clear responsibility and accountability, victims would be left without an effective remedy for the harm they experienced, and parties to a conflict would enjoy impunity for attacks by fully autonomous weapon systems, and would, hence, have less incentive to behave ethically and in compliance with the law.

Fully autonomous weapons also pose a pressing danger to international peace and security. A robotic arms race, for instance, is a real possibility. It is estimated that more than 70 countries have acquired drone technology, and a handful, including China, Israel, Russia, the United Kingdom, and the United States, have armed drones and other robotics technology. If one or more of these States chose to deploy fully autonomous weapons others may feel compelled to abandon policies of restraint.

We should also expect fully autonomous weapons technologies to proliferate. Weapons systems with a high degree of autonomy are vulnerable to being appropriated and hacked. They could be intercepted and misused by third parties, and it cannot be excluded that non-State actors could gain access to such technologies.

Moreover, scientists affiliated with the International Committee for Robot Arms Control (ICRAC), a founding member of the campaign, warn that it is highly uncertain how devices controlled by complex algorithms will interact. Such interactions could create unstable and unpredictable behavior, behavior that could initiate or escalate conflicts, or cause unjustifiable harm. A United States Department of Defense Directive on ‘Autonomy in Weapons Systems’ issued in November last year acknowledges the dangers of failures, unintended engagement or loss of control of the system to unauthorized parties.[2]

Aside from these significant humanitarian, legal and security challenges, the overriding consideration in relation to fully autonomous weapons systems is that allowing machines to make the decision to kill a human being crosses a fundamental moral line. There is strong international consensus that not all weapons area acceptable, and we believe that giving machines the power to choose who lives and dies on the battlefield is an unacceptable application of technology. As a retired United States Major General recently put it: ‘death by algorithm is the ultimate indignity’.[3]

In our view, meaningful human control of any autonomous weapon system and accountability for their use are essential to ensuring both humanitarian protection and the rule of law. Meaningful human control requires active cognitive participation of a human being in every individual attack and sufficient time for deliberation on the nature and significance of a target, its context and the anticipated effects of an attack. However, in view of the demands within the military for increasingly rapid response times, and given the significant investments into automatic, and increasingly autonomous weapons, there is grave concern that if this development is left unchecked, robotic weapons could make life and death decisions on the battlefield with no more than a veneer of human control.

To ensure that weapons remain under meaningful human control, there is an urgent need to draw clear lines against the development of fully autonomous weapons. The Campaign to Stop Killer Robots welcomed the recent statement of the High Representative for Disarmament Affairs, Angela Kane, to the Human Rights Council, in which she noted that ‘the pace of technological developments continues to accelerate’, and ‘So too must our efforts to come to grips with the implications of emerging weapon systems’.[4]

Agreement is needed to establish controls on these weapons before investments, technological momentum, and new military doctrine make it difficult to change course. We agree with the High Representative that ‘we need not wait for a weapon system to fully emerge before appropriate action can be taken to understand its implications and to adopt effective measures’. Exploding bullets were banned in 1868 and blinding laser weapons in 1995, before a stream of victims gave visible proof of their unacceptable effects. Experience with landmines and cluster munitions shows that waiting for widespread use of weapons that cause unacceptable humanitarian harm can result in crises that are difficult and expensive to resolve.

The Campaign to Stop Killer Robots therefore supports the UN Special Rapporteur’s call for national moratoria on robotic weapons systems that can select and engage targets without further intervention by a human. A moratorium should be seen as a first step towards a comprehensive, pre-emptive ban on the development, production and use of fully autonomous weapons. This could be achieved through an international treaty, as well as through national laws and other measures.

Significant military states recognize the importance of addressing this issue. The campaign is calling on States to elaborate and publicly articulate their policies on autonomous weapons, particularly with respect to the ethical, legal, policy, technical, and other concerns that have been raised. Existing policy formulations of the United States and the United Kingdom are positive steps in that direction, but these efforts do not clearly indicate where the line should be drawn against fully autonomous weapons systems – and they still leave the door open to adoption of such systems in the future.

A collective effort is needed to curb the development toward greater autonomy in weapons systems, before we find the matter has, literately, been taken out of human hands. Fully autonomous weapons systems raise a broad range of concerns that States could come together to discuss in a variety of international fora. During last month’s debate in the Human Rights Council many States recognized that fully autonomous weapons systems merit special consideration and action, including within the appropriate United Nations fora. We noted that several states suggested that the Convention on Certain Conventional Weapons (CCW) should look at these weapons. Another avenue could be for the UN General Assembly to call for a UN study on the subject. In the past, such studies have helped to gather momentum and resulted in meaningful action.

The Campaign to Stop Killer Robots seeks to further policy debate at the national and international levels. Events, such as the one organized by the UN Institute for Disarmament Research on ‘Lethal Autonomous Robotics’ (23 May 2013) can make an important contribution to furthering this debate. We also welcome the recommendation of the Special Rapporteur to establish a High Level Panel consisting of experts from different fields to consider how to effectively address this challenge through a broad-based international dialogue, and we would support the work of that panel in any way possible.

With a view to the control of emerging weapon technologies more broadly we endorse the Special Rapporteurs call for more transparency in States’ weapons review processes. Promotion of transparency and accountability in relation to the development, proliferation and use of fully autonomous weapons, and weapons generally, and increased public scrutiny of how determinations about the acceptability of weapons are made, are central to the prevention of unacceptable harm, and in order to better understand and address in a timely manner the disarmament and security challenges raised by emerging weapon technologies.

To conclude, the Campaign to Stop Killer Robots believes that national and international action must be taken to prevent fully autonomous weapons from being deployed and we urge UN actors to continue and strengthen their efforts, in partnership with States, international organisations and civil society, to bring that action about.

[4] Statement of the High Representative for Disarmament Affairs to the 23rd session of the Human Rights Council, on the topic of lethal autonomous robotics, delivered on behalf of the High Representative by Mr. Jarmo Sareva, Director of the Geneva Branch of UNODA, http://www.un.org/disarmament/update/20130530/.