(CNN) -- Two of Paraguay's most important neighbors, Brazil and Uruguay, said Sunday they were pulling their ambassadors from the country in the wake of the impeachment of Paraguay's president.

Brazil's foreign ministry said the action was "due to the breakdown of democracy in Paraguay," and vowed to take up the impeachment with the regional blocs Mercosur and Unsur.

The moves follow the lightning-quick removal from office of President Fernando Lugo on Friday.

The impeachment procedures appear to have been carried out in accordance with the Paraguayan Constitution, but some Latin American presidents are calling it a coup d'etat and refuse to recognize the new president, Federico Franco.

Lugo went from president to disgraced leader in less than 48 hours.

The former Catholic bishop was unpopular with lawmakers, and many had concerns about his credibility after he he admitted to fathering at least two children while still in the priesthood. In all, four women claim they had babies by Lugo while he was bishop.

But nine days ago, there was no reason to believe that Lugo would find himself out of a job before his term ended in August 2013.

Then Paraguay impeached its president so fast that its neighbors needed a few days to figure out how to react.

By Sunday, Brazil and Uruguay were expressing deep concerns.

"The Brazilian government condemns the expedited removal" of the president, the foreign ministry said in a statement.

"The imposition of a new president under these conditions is not consistent with fundamental democratic practices that we must preserve in the region," the office of Uruguay's President Jose Mujica said via the foreign ministry.

There had been calls in the past for the impeachment of Lugo, but the scandals did not rock his position too much.

That changed on June 15, when police and landless peasants clashed in eastern Paraguay, resulting in 17 deaths.

Peasants fired on police who were trying to evict them from private property, initiating the deadly confrontation, local authorities and state-run media said.

The violence occurred in Curuguaty, a remote community about 240 kilometers (150 miles) northeast of the Paraguayan capital, Asuncion, near the Brazilian border.

In response to the incident, Lugo replaced his national police chief and interior minister. The country's liberal party, which backed Lugo, were incensed to learn that the new minister was from another party.

As the outcry over the deadly clash continued, the liberal party announced Thursday that it was withdrawing its support of Lugo, and an impeachment vote was heard in the lower chamber of congress that day. The vote in favor of impeachment was 76-1.

The next day, Lugo's defense team had two hours to defend Lugo from what it call vague charges of incompetence. The Senate impeached the president in a 39-4 vote.

That was it.

Lugo said Paraguayan history and democracy had been "deeply wounded."

Other Latin American countries expressed concern.

"If indeed it is recognized that the impeachment process happened according to the procedure established in the Paraguayan Constitution, Mexico considers that the process did not offer President Lugo the space and time for a proper defense," Mexico's foreign ministry said.

The presidents of Venezuela, Argentina, Bolivia, Ecuador and the Dominican Republic said they will not recognize Franco's administration, and Argentina on Saturday withdrew its ambassador to Paraguay.

Brazil and Uruguay followed suit on Sunday.

Bolivia "will not recognize a government that does not rise from the ballot box and the will of the people," Bolivian President Evo Morales said.

Venezuelan President Hugo Chavez said, "I have no doubt that President Fernando Lugo was not given a right to his defense, it happened from one day to the next, and the sentence was decided before the trial."

As of Friday night, not a single international leader had called him to congratulate him on his being sworn in as president, Franco told CNN en Español.

"Of course it worries me. I am aware that I am assuming the presidency in an unfavorable condition," he said.

Franco reiterated that the impeachment happened within the parameters of the constitution, and pointed to the large margins by which Lugo was voted out of office.

"It means that here we have a unanimous position. The Paraguayan people, I think, are satisfied with this decision," he said.

Despite the cold shoulder from neighboring countries, Franco said he will reach out and explain the legality of Lugo's ouster and seeks to have good international relations.

His goal is for Paraguay to be recognized internationally by the time he hands the government over to the next president next year, he said.

"I have the duty and the responsibility to initiate a process that the next government can continue," he said, especially when it comes to important domestic issues such as security and family agriculture.

Most of the countries opposed to Franco's presidency belong to the Union of South American Nations, known as UNASUR. The foreign ministers of that regional body were in Paraguay to study the issue.

The UNASUR ministers "have an attitude of respect for the sovereignty of Paraguay, but also an attitude of respect for democracy," Brazilian President Dilma Rousseff said.

Brazil is one of Paraguay's most important neighbors, and Rousseff declined to give a strong position one way or another over the recent events.

"From this situation I am sure there will be a consequence," is all she said.

In a news conference Saturday, Franco said he hopes relations with Brazil will remain "harmonious," according to Brazil's state-run Agencia Brasil news agency.

The United States also weighed in a more neutral manner.

"We urge all Paraguayans to act peacefully, with calm and responsibility, in the spirit of Paraguay's democratic principles," State Department spokeswoman Darla Jordan said.

Online Public Information File

Viewers with disabilities can get assistance accessing this station's FCC Public Inspection File by contacting the station with the information listed below. Questions or concerns relating to the accessibility of the FCC's online public file system should be directed to the FCC at 888-225-5322, 888-835-5322 (TTY), or fccinfo@fcc.gov.