Senior figures inside British intelligence have been alarmed by GCHQ's secret decision to tap into transatlantic cables in order to engage in the bulk interception of phone calls and internet traffic.

According to one source who has been directly involved in GCHQ operations, concerns were expressed when the project was being discussed internally in 2008: "We felt we were starting to overstep the mark with some of it. People from MI5 were complaining that they were going too far from a civil liberties perspective … We all had reservations about it, because we all thought: 'If this was used against us, we wouldn't stand a chance'."

The Guardian revealed on Friday that GCHQ has placed more than 200 probes on transatlantic cables and is processing 600m "telephone events" a day as well as up to 39m gigabytes of internet traffic. Using a programme codenamed Tempora, it can store and analyse voice recordings, the content of emails, entries on Facebook, the use of websites as well as the "metadata" which records who has contacted who. The programme is shared with GCHQ's American partner, the National Security Agency.

Interviews with the UK source and the NSA whistleblower Edward Snowden raise questions about whether the programme:

■ Exploits existing law which was passed by parliament without any anticipation that it would be used for this purpose.

■ For the first time allows GCHQ to process bulk internal UK traffic which is routed overseas via these cables.

■ Allows the NSA to engage in bulk intercepts of internal US traffic which would be forbidden in its own territory.

■ Functions with no effective oversight.

The key law is the Regulation of Investigatory Powers Act 2000, Ripa, which requires the home secretary or foreign secretary to sign warrants for the interception of the communications of defined targets. But the law also allows the foreign secretary to sign certificates that authorise GCHQ to trawl for broad categories of information on condition that one end of the communication is outside the UK.

According to the UK source: "Not so long ago, this was all about attaching crocodile clips to copper wires. And it was all about voice. Now, it's about the internet – massive scale – but still using the same law that was devised for crocodile clips. Ripa was primarily designed for voice, not for this level of interception. They are going round Ripa. The legislation doesn't exist for this. They are using old legislation and adapting it."

The source claimed that even the conventional warrant system has been distorted – whereas police used to ask for a warrant before intercepting a target's communications, they will now ask GCHQ to intercept the target's communications and then use that information to seek a warrant.

There is a particular concern that the programme allows GCHQ to break the boundary which stopped it engaging in the bulk interception of internal UK communications. The Ripa requirement that one end of a communication must be outside the UK was a significant restriction when it was applied to phone calls using satellites, but it is no longer effective in the world of fibre-optic cables. "The point is that this is an island," the source said. "Everything comes and goes – nearly everything – down fibre-optic cables. You make a mobile phone call, it goes to a mast and then down into a fibre-optic cable, under the ground and away. And even if the call is UK to UK, it's very likely – because of the way the system is structured – to go out of the UK and come back in through these fibre-optic channels."

Internet traffic is also liable to be routed internationally even if the message is exchanged between two people within the UK. "At one point, I was told that we were getting 85% of all UK domestic traffic – voice, internet, all of it – via these international cables."

Last year, the government was mired in difficulty when it tried to pass a communications bill that became known as the "snoopers' charter", and would have allowed the bulk interception and storage of UK voice calls and internet traffic. The source says this debate was treated with some scepticism inside the intelligence community – "We're sitting there, watching them debate the snoopers' charter, thinking: 'Well, GCHQ have been doing this for years'."

There are similar concerns about the role of the NSA. It could have chosen to attach probes to the North American end of the cables and documents shown to the Guardian by Edward Snowden suggest that key elements of the Tempora filtering process were designed by the NSA. Instead, the NSA agency has exported its computer programs and 250 of its analysts to operate the system from the UK.

Initial inquiries by the Guardian have failed to explain why this has happened, but US legislators are likely to want to check whether the NSA has sought to bypass legal or policy requirements which restrict its activity in the US. This will be particularly sensitive if it is confirmed that Tempora is also analysing internal US traffic.

The UK source challenges the official justification for the programme; that it is necessary for the fight against terrorism and serious crime: "This is not scoring very high against those targets, because they are wise to the monitoring of their communications. If the terrorists are wise to it, why are we increasing the capability?

"The answer is that you can't stop it. It is a self-fulfilling prophecy. The more we develop communications technology, the more they develop technology to intercept it. There was MS Chat – easy. Then Yahoo chat – did that, too. Then Facebook. Then Skype. Then Twitter. They keep catching up. It is good for us, but it is bad for us."

It is clear from internal paperwork that GCHQ has created systems to restrain the use of this powerful tool and to ensure that its use complies not only with Ripa but also with the 1998 Human Rights Act, which requires essentially that the use of the data must be proportional to the crime or threat investigated. Defenders insist that the mass of data is heavily filtered by the programme so that only that relating to legitimate targets is analysed.

However, there are doubts about the effectiveness of this. First, according to the UK source, "written definitions for targeting and filtering are very elastic. They are wide open to interpretation." The target areas defined by the Ripa certificates are secret.

Second, there is further room for interpretation when human analysts become involved in using the filtered intelligence to produce what are known as "contact chains". "Here is target A. But who is A talking to? Now we're into B and C and D." If analysts believe it is proportional, they can look at all the traffic – content and metadata – relating to all of the target's contact." GCHQ audits a sample of its analysts' work – believed to be 5% every six months – but even the statistical results of these audits are also secret.

Beyond the detail of the operation of the programme, there is a larger, long-term anxiety, clearly expressed by the UK source: "If there was the wrong political change, it could be very dangerous. All you need is to have the wrong government in place. It is capable of abuse because there is no independent scrutiny."