(CNN) -- Gov. David Paterson had no intention of appointing Caroline Kennedy to fill the Senate seat vacated by Hillary Clinton, a source close to the New York governor told CNN Thursday.

Caroline Kennedy has withdrawn her name from consideration for Hillary Clinton's Senate seat.

The source told CNN that Paterson did not think Kennedy was "ready for prime time," citing her efforts, at times awkward, to try to win the appointment.

"She clearly has no policy experience and couldn't handle the pressure," said the source, who spoke on the condition of anonymity because of the sensitivity of the situation. "Why would he pick her given how badly she handled herself in recent weeks?"

Kennedy, 51, cited personal reasons for her decision not to continue pursuing the Senate seat. She informed the media at midnight, as Wednesday turned to Thursday.

Paterson is charged with naming a replacement for Clinton, who resigned her seat to become the secretary of state in President Obama's administration. Paterson will appoint someone to hold the seat until a new election is held in 2010.

A Kennedy ally, though, denied Kennedy had any indication Paterson was leaning against choosing her to fill out Clinton's term.

And another Kennedy confidante said Kennedy allies are getting frustrated about what they perceive as the governor's insiders slighting her.

Paterson, who is expected to name his choice Friday, has been coy about who he will pick to replace Clinton. Kennedy had been very public in expressing her interest in the seat by meeting with state and community leaders throughout the state.

Her uncle, Sen. Edward Kennedy, D-Massachusetts, was rushed to the hospital on Tuesday following Obama's swearing-in ceremony. But an aide to Caroline Kennedy indicated Sen. Kennedy's health was not the reason she decided to withdraw.

By early Thursday afternoon, there was still confusion over how and why Kennedy withdrew her name from consideration. The source close to the governor told CNN that Kennedy initially called Paterson in the afternoon to tell him that "she was feeling unsure about her candidacy" and "asked him for 24 hours to think about it."

A Kennedy aide said that during the call with the governor, it was Paterson who told her to take more time to think about her decision.

Don't Miss

Later Wednesday evening, the governor started getting phone calls from the media saying they heard she had withdrawn. At around 11 p.m. ET, the source close to the governor said Kennedy called Paterson and "said she was issuing a statement saying she was in the race." One hour later, the source said Kennedy issued a statement withdrawing from the race.

Paterson's office finally addressed the matter late Thursday afternoon in a prepared statement.

"Last night Caroline Kennedy informed the governor that she was withdrawing her name from consideration for an appointment to the United States Senate for personal reasons," Paterson's office said. "This decision was hers alone."

Paterson's office acknowledged the Wednesday afternoon conversation between the governor and Kennedy, but did not divulge what was said.

"Out of respect for her decision making process, the governor's Office did not respond to any inquiries in order to allow her time to deliberate," Paterson's office said in the statement.

"The governor considers Caroline a friend and knows she will continue to serve New York well inside or outside of government. We wish her well in all her future endeavors."

Paterson's office also noted that he is "now entering the final phase of his selection process."

Kennedy was widely considered the front-runner to replace Clinton, who won the New York Senate seat in 2000, and almost all of the buzz about a replacement centered around her. But as the weeks dragged on, polling showed her popularity among New Yorkers dwindling.

A poll earlier this month suggested New Yorkers preferred Attorney General Andrew Cuomo, son of former New York Gov. Mario Cuomo, to Kennedy, with 31 percent of respondents favoring Cuomo and 24 percent preferring Kennedy.

However, "celebrity magazines and cocktail party chatter don't build support for senators," Democratic national committeeman and CNN contributor Robert Zimmerman said Thursday.

Zimmerman was one of the first New York Democrats to say Kennedy wouldn't get the Senate seat and that Paterson wouldn't bow to pressure. Her handlers "didn't demonstrate they had a real plan of action," he said Thursday.

"The disorganized way she dropped out was reflective of the very disorganized campaign she pursued in seeking the Senate seat," he added.

New York Reps. Carolyn Maloney, Kirsten Gillibrand and Steve Israel have also been mentioned as possible replacements for Clinton. All three drew single-digit support in the January poll.

Paterson's office also said the governor had not obtained any information during the selection process that might prompt a candidate to withdraw. There has been much speculation as to why Kennedy, a member of Obama's vice presidential search committee, withdrew her name.

Despite the governor's public statement, a source close to Paterson told CNN that Kennedy did have potentially embarrassing issues including problems with her taxes and with her nanny. People who move in New York political circles were buzzing Thursday about the endless finger pointing between people loyal to Paterson and those loyal to Kennedy.

Contacted by CNN to address the allegations of problems with taxes and a nanny, Kennedy's spokesman Stefan Friedman said in a prepared statement: "Caroline Kennedy withdrew her name for consideration from the United States Senate for personal reasons. Any statements to the contrary are false. The governor set up a fair and deliberative selection process. This kind of mudslinging demeans that process and all those involved."