(CNN)-- U.S. Defense Secretary Leon Panetta warned Thursday that the United States was "reaching the limits of our patience" with Pakistan over militants that attack U.S. forces in neighboring Afghanistan from havens within its borders.

"It's extremely important that Pakistan take action to prevent this kind of safe haven," he said during an unannounced visit to Kabul, and that militants cannot use the country as a "safety net" from which to attack U.S. soldiers.

"We have made that very, very clear time and time again, and we will continue to do that," he said.

Panetta also stressed that troops on the Afghan border have every right to defend themselves against the al Qaeda-linked Haqqani network and other militant groups when they launch attacks from Pakistan.

"Anybody who attacks U.S. soldiers is our enemy. We are not going to take it," he said.

"We have every responsibility to defend ourselves, and we are going to make very clear that we are prepared to take them on -- and we have to put pressure on Pakistan to take them on as well."

Afghan Defense Minister Gen. Abdul Rahim Wardak, speaking alongside Panetta at a news conference, said havens in Pakistan were the "most important" issue confronting Afghanistan as it seeks to secure a stable future.

"This kind of public messaging from a senior member of the U.S. administration is taken very seriously in Pakistan and reduces the space for narrowing our bilateral differences at a critical time in the negotiations," she said.

"It adds an unhelpful twist to the process and leaves little oxygen for those of us seeking to break a stalemate."

Islamabad has yet to respond to Panetta's remarks.

The defense secretary may also have irritated Pakistani leaders with remarks during a stop in India on Wednesday in which he joked about Pakistan being kept out of the loop as U.S. intelligence tracked al Qaeda founder Osama bin Laden to a Pakistani town, flew undetected into the country and killed him.

Asked if Pakistan had interfered in the operation in any way, Panetta replied to laughter: "They didn't know about our operation. That was the whole idea."

A senior military official in Pakistan, who asked not to be named because he is not authorized to speak to the media, told CNN: "This is Washington's way of piling on pressure on Pakistan. It's a tactic they've used for a long time. Pakistan has sacrificed thousands of lives, and here you have someone laughing. It shows their level of thinking."

Panetta's stern words came three days after a CIA drone strike in Pakistan's tribal region killed Abu Yahya al-Libi, the No. 2 man in al Qaeda and a longtime public face of the terrorist network, and at a time of strained Pakistani-U.S. relations.

Monday's strike was the third such deadly attack in as many days and the 21st suspected U.S. drone strike in Pakistan this year. At least six missiles were fired at a militant compound near the town of Mir Ali in the North Waziristan region, near the Afghan border.

Pakistani lawmakers have called for an immediate end to the drone strikes, which have been denounced for killing civilians.

While in India, Panetta said Washington had made clear to Pakistan that it would defend itself against those who sought to attack the United States.

"And we have done just that. We have gone after their leadership, and we have done it effectively ... targeting al Qaeda leadership and terrorists.

"And very frankly, the terrorists who threaten the United States threaten Pakistan as well. This is not just about protecting the United States; it's also about protecting Pakistan. And we have made very clear that we are going to continue to defend ourselves."

Is not planning for more defense cuts the right plan?

Panetta is not the first high-ranking U.S. official to point the finger at Pakistan over a failure to take on extremists within its borders.

Addressing Congress in September, just before he retired as chairman of the Joint Chiefs of Staff, Adm. Mike Mullen said the Haqqani network acted as a "veritable arm" of Pakistan's Inter-Service Intelligence agency.

He also told CNN that elements in Pakistan's spy service were "very active" with the Haqqani network in launching attacks on U.S. forces in Afghanistan and that the U.S. and Pakistani governments and military must work together to tackle the problem.

Mullen's blunt remarks to U.S. senators were widely interpreted as an expression of the growing impatience of U.S. officials with Pakistan's unwillingness to stop the attacks and the belief that elements in the Pakistani government are actively supporting the insurgents.

CNN national security analyst Peter Bergen said that Panetta's criticism did not represent a significant change of tone and that Pakistani officials mostly "tune out" such comments.

"They are usually pretty frustrated by calls from the United States to do more when they know that more Pakistani soldiers have died fighting the Taliban than U.S. and NATO soldiers combined," he said.

At the same time, the Pentagon is very frustrated over the continued presence of the Haqqani network, since it is the most effective of the militant groups operating out of Pakistan, Bergen said.

But Pakistan is not going to move against the Haqqani network for multiple reasons, Bergen said, including the risk that it wouldn't go well from a military point of view and the fact that the United States plans to withdraw the bulk of its forces from Afghanistan by the end of 2014.

However, while Pakistan is keen to ensure that India does not gain sway in Afghanistan after the withdrawal of international forces, it is also in Islamabad's interests to see its neighbor remain stable and out of the hands of the Taliban, he added.

Panetta visits former U.S. base in Vietnam

Panetta did not visit Islamabad, the Pakistani capital, during his nine-day Asia trip, which included stops in Vietnam and Singapore as well as New Delhi.

Relations between Washington and Islamabad have been particularly tense since NATO airstrikes killed two dozen Pakistani soldiers in November.

After the fatal airstrikes, the Pakistani government shut down the two NATO supply routes in the country, asked the United States to vacate an air base on its territory and boycotted a conference about the future of Afghanistan. NATO insists that the incident was an accident.

U.S. defense secretary announces new strategy with Asia

President Barack Obama's administration recently defended its use of unmanned drones to target suspected terrorists overseas in a rare public statement, with John Brennan, the president's top counterterrorism adviser, saying the strikes are conducted "in full accordance with the law."

The program uses unmanned aerial vehicles, often equipped with Hellfire missiles, to target suspected terrorists in remote locations overseas, with many such strikes occurring in Yemen and Pakistan, despite internal opposition to the practice within the latter country.

Brennan said the United States "respects national sovereignty and international law" and is guided by the laws of war in ordering those attacks.

Despite the public discussion of bad blood between the United States and Pakistan, Bergen cautions against reading too much into the dispute over NATO ground supply routes. Pakistan's airspace has remained open to NATO forces throughout the dispute and is much more important to international operations, he said.

Pakistan was also invited to last month's NATO summit in Chicago -- at which leaders signed off on the plan for withdrawal of international forces from Afghanistan -- without any insistence that the supply lines be opened, he said, in a small sign of easing tensions.

The Pakistani border area is widely believed to be the operating base for the Haqqani network and other militant groups that have attacked international troops in neighboring Afghanistan.

Online Public Information File

Viewers with disabilities can get assistance accessing this station's FCC Public Inspection File by contacting the station with the information listed below. Questions or concerns relating to the accessibility of the FCC's online public file system should be directed to the FCC at 888-225-5322, 888-835-5322 (TTY), or fccinfo@fcc.gov.