WASHINGTON (CBS DC) — U.S. Defense Secretary Chuck Hagel stated there is a “real possibility” that there will be a total troop pullout from Afghanistan if President Hamid Karzai does not sign a security pact.

Speaking with CBS News State Department correspondent Margaret Brennan on Sunday’s “Face the Nation,” Hagel said that the potential total pullout now rests in the hands of the Afghan president.

“It’s a very real possibility because if we don’t have a bilateral security agreement, which I have noted, that means we can’t protect our forces which would be here after 2014,” said Hagel. “No international partners will come. Afghanistan essentially will be alone. But we have no other option.”

Just a day after meeting with officials in Kabul, Brennan asked Hagel if the U.S. had miscalculated in thinking Karzai could be a partner. Although the bilateral security agreement was approved by members of other top Afghan officials, Karzai began making additional demands in order to sign the agreement.

Hagel said the U.S. was “surprised” by Karzai’s refusal to sign the agreement, but still views the Afghan president as a partner to the U.S.

In a question directed at the use of the word “retreat” to describe the possible troop pullout from the country the U.S. has occupied since 2001, Hagel stated that “you can use any term you want, ‘retreat’ or ‘not renewing our efforts here post-2014’ – you can say it any way you want.”

“But what I’m saying is, unless we have the security of an agreement to protect our forces, then we’ll have no choice – we will not be able to stay.”

Hagel told Brennan that a debate should take place about whether such an agreement or continued stay is “worth it,” and Hagel stated that “it is now up to President Karzai to make a decision.”