Leon Panetta has spent more than 40 years in Washington politics. He's worked as a congressman, the White House chief of staff, the director of the CIA and, most recently, the secretary of defense.

Cliff OwenAP

For more than 40 years, Leon Panetta has split his life on two coasts: his home in California and his work in Washington, D.C. It's a career that included 16 years in Congress, stints as White House chief of staff for President Clinton, and as the head of the CIA and the Pentagon under President Obama.

As Panetta prepares to leave his job as defense secretary, he sat down with Rachel Martin, host of Weekend Edition Sunday, to talk about his years in Washington and serving in the Obama administration.

During the past two years at the Pentagon, Panetta has overseen the aftermath of the war in Iraq, the drawdown of troops from Afghanistan and a massive budget battle. If Congress doesn't reach a deficit deal, the Pentagon could be forced to cut close to $50 billion this year.

It's the kind of battle Panetta has fought before, having participated in almost every major summit on budget issues since the Reagan administration. But he says there's been a big change since then.

"One of the differences is that the leadership was very strong about demanding that both sides sit down and not come out of a room until they had found a compromise," Panetta says.

Heading The CIA

Budget issues were the kinds of problems that Panetta dealt with for a good part of his political career, but when Obama asked him to head the CIA, the issues he would grapple with would be much different.

Panetta was home in California when he got the call from the White House chief of staff; he was initially skeptical of taking the job.

"I said you've got to be out of your mind," Panetta says.

But Panetta says Obama thought he could help restore trust in the CIA, a daunting task, as trust in the agency had eroded, he says, since the attacks of Sept. 11, 2001.

"In many ways, I'm a sucker for a challenge," he says.

As head of the CIA, Panetta oversaw the U.S. drone campaign against al-Qaida. When the intelligence identified the location of a particular target, Panetta often had to decide whether to fire.

"As a Catholic, suddenly realizing I had the responsibility of saying, 'We're going to go after somebody,' it was something I did not take lightly," he says. "It's a lot of responsibility."

Panetta justifies the use of drones as a necessary tool to "keep the country safe," which he says was his mission both as head of the CIA and as defense secretary.

"I felt they were legitimate if we followed the law, and did what we had to do under our law, to make sure that we were being true to the American people with regard to our responsibility," he says.

There is wide uncertainty about how many civilians have been killed in U.S. drone strikes. One estimate from the New America Foundation says around 300 civilians were killed in Pakistan from 2004 to the beginning of this year.

There is at least one case where U.S. officials, including Panetta, knew that a woman was present at a possible strike site, and the attack was ordered anyway. A U.S. official told NPR a strike with non-combatants in the area would only happen in "exceptional circumstances against very high-level terrorists."

Changing The Pentagon

At the CIA, Panetta saw close up how the government exercises power in the name of national security. He saw the same thing at the Pentagon, but as defense secretary he got to use his own power to change the institution.

In January, Panetta announced an end to a 1994 policy that banned women from serving in combat. Panetta says, for him, the issue was personal.

"I'm the son of immigrants, and in many ways I've lived the American dream," he says. "It is about giving everybody the opportunity to succeed. There's no guarantees, but everybody ought to have a chance."

After Panetta's decision, there has been concern from some in the military of standards, in particular the idea that the standards of some jobs might have to be lowered to accommodate women. He says it is simply a chance for the military to re-evaluate its standards and make improvements.

"Every profession that's gone through this ... have had to look at their standards and really determine 'are these really the standards that we need in order to have a good fireman [or] good policeman,'" he says. "I think the military is going to have to go through the same process."

A Californian At Heart

With every job he's had in Washington, Panetta has made it a priority to make it home to Monterey, Calif., as often as he can to see his family. The reason this was so important to him, he says, is that he feels it's good for those who work in Washington to get out of the city.

"The problem with Washington is that you can become very confined; you lose your perspective," he says. "Throughout my career, I've always thought it was important to have a chance to be just another citizen, because that ... energizes you to be able to come back and make the kind of decisions you have to make."

On the walls of Panetta's office, a long row of clocks show the time in Kabul, Baghdad and Washington. But the watch Panetta wears on his wrist is set three hours behind — California time.