US secretary of state John Kerry made a surprise visit to Kabul hours after the US military handed over a disputed prison to Afghan control, reviving hopes that the often fractious allies may be able to smooth ties ahead of a crucial election and the departure of Nato troops.

Kerry, who has a long – and, by US standards – unusually positive relationship with Karzai, said he was "on the same page" as the Afghan leader when it came to peace talks with the Taliban, and added that he was impressed by Afghan efforts to ensure a "safe, secure" presidential vote next spring.

Karzai cannot stand for election again, so the country will get a new leader after more than a decade under the rule of a man who went from darling of the US establishment in the days following the overthrow of the Taliban to a reluctant partner and now sometime antagonist.

Kerry's smooth visit came just a few days after a tumultuous trip by new defence secretary Chuck Hagel. The planned handover of Bagram prison during that visit was cancelled after Karzai said he would release some prisoners, and then a joint press conference was called off after the Afghan leader suggested the Taliban were colluding with the US.

But in Monday's cheerful, hour-long joint meeting with the media, Karzai appeared to back away from those comments, and added that "today was a very good day".

Hours earlier, the top US and Nato commander in Afghanistan, General Joseph Dunford, handed over the Bagram jail after signing an agreement with Afghan defence minister Bismillah Khan.

The deal was believed to contain details on the future detention of high-profile Taliban prisoners, although unlike an earlier agreement it will not be made public. Dunford said the transfer was a milestone in Afghan efforts to take control of their own security, after more than a decade in which foreign troops have led the fight against the Taliban.

The US military alone expects to spend about $6bn bringing equipment and vehicles home ahead of a 2014 deadline for Nato combat troops to leave. Some US soldiers are expected to stay on to fight groups like al-Qaida and train the Afghan army, but Washington has said they will no longer take on the Taliban, leaving that to Afghan forces.

"This ceremony highlights an increasingly confident, capable and sovereign Afghanistan," Dunford told assembled officials, soldiers and journalists at the jail, although the US military has promised to provide advisors and nearly $40m to support the complex that it built.

Karzai had demanded control of the jail for many years, saying the US detention of thousands of Afghan citizens in their own country was a serious violation of national sovereignty.

But the US was concerned about a pattern of insurgents released from jail rejoining the fight against Afghan and Nato forces. Zakir Qayyum a former Guantánamo detainee released into Afghan custody in 2007 was one such case, the Associated Press reported. He was freed four months later and rejoined the Taliban.

They may have set aside those concerns to help smooth negotiations on a bilateral security agreement that would allow US forces to stay on in Afghanistan past 2014. Boots on Afghan soil would help Washington keep an eye on al-Qaida linked militants, and monitor Iranian and Pakistani efforts to influence their weaker and poorer neighbour.

Negotiations were reported to have all but stalled before the prison handover. "It's all part of the bilateral security agreement; it's about a shift that's going on in how the US is looking at what's important," one US official knowledgeable about detention issues told the New York Times. "We have to look at the larger picture: what's the US strategic interest here?"