GCHQ, the government's monitoring agency, acted illegally by developing spy programs that remotely hijack computers' cameras and microphones without the user's consent, according to privacy campaigners.

A legal challenge lodged on Tuesday at the investigatory powers tribunal (IPT) calls for the hacking techniques – alleged to be far more intrusive than interception of communications – to be outlawed. Mobile phones were also targeted, leaked documents reveal.

The claim has been submitted by Privacy International following revelations by the whistleblower Edward Snowden about the mass surveillance operations conducted by GCHQ and its US counterpart, the National Security Agency (NSA).

The 21-page submission details a host of "malware" – software devised to take over or damage another person's computer – with such esoteric names as Warrior Pride, Gumfish, Dreamy Smurf, Foggybottom and Captivatedaudience.

More intrusively, Privacy International alleges, the programs enable surveillance of any stored content, logging of keystrokes and "the covert and unauthorised photography or recording of the user and those around him". It is, the claim maintains, the equivalent of "entering someone's house, searching through his filing cabinets, diaries and correspondence, and planting devices to permit constant surveillance in future, and, if mobile devices are involved, obtaining historical information including every location he had visited in the past year".

Such break-ins also leave devices vulnerable to attack by others "such as credit card fraudsters, thereby risking the user's personal data more broadly", Privacy International argues. "It is the modern equivalent of breaking in to a residence, and leaving the locks broken or damaged afterwards."

The claim acknowledges that it is unclear how many computers or mobiles have been infected but points out that leaked documents show the agencies have the ability to scale up the programme to infect millions of computers and devices around the world.

GCHQ itself had reservations about the legality of these surveillance operations, Privacy International claims, pointing to a leaked document noting that "continued GCHQ involvement may be in jeopardy due to British legal/policy restrictions".

The activities of GCHQ breach the right to private and family life under article 8 of the European convention on human rights and the 1990 Computer Misuse Act, Privacy International alleges.

The submission states: "Privacy International accepts that, in principle, surveillance may be conducted for legitimate aims such as national security. The issue is therefore whether the interference is 'in accordance with the law' or 'prescribed by law', and whether it is necessary and proportionate."

The IPT is a partially secret court which investigates complaints about MI5, MI6, GCHQ and the use of surveillance powers by government, police and local authorities. Many of its hearings take place behind closed doors.

The IPT has extensive powers to demand to see all relevant intelligence and evidence. Often complainants are told the tribunal can neither confirm nor deny whether surveillance has taken place.

This latest submission will join a long line of legal challenges brought by civil liberties groups following Snowden's revelations. The IPT is already looking at complaints over GCHQ and NSA use of mass interception programmes such as Prism and Tempora. Other claims have been lodged against other European governments, at the European court of human rights and with the Organisation for Economic Cooperation and Development (OECD), alleging abuse of telecommunications equipment.

Eric King, deputy director of Privacy International, said: "The hacking programmes being undertaken by GCHQ are the modern equivalent of the government entering your house, rummaging through your filing cabinets, diaries, journals and correspondence, before planting bugs in every room you enter. Intelligence agencies can do all this without you even knowing about it, and can invade the privacy of anyone around the world with a few clicks.

"All of this is being done under a cloak of secrecy without any public debate or clear lawful authority. Arbitrary powers such as these are the purview of dictatorships, not democracies. Unrestrained, unregulated government spying of this kind is the antithesis of the rule of law and government must be held accountable for their actions."

In the past GCHQ has declined to comment on any of its specific programmes, but stressed that its activities are proportional and comply with UK law. Responding to allegations earlier this year that mobile phones were being targeted through downloaded apps, the agency said: "It is a longstanding policy that we do not comment on intelligence matters."

A spokesman added: "Furthermore, all of GCHQ's work is carried out in accordance with a strict legal and policy framework that ensures that our activities are authorised, necessary and proportionate, and that there is rigorous oversight, including from the secretary of state, the interception and intelligence services commissioners, and the parliamentary intelligence and security committee. All our operational processes rigorously support this position."