On the roof of a partly-finished building that overlooks the former home of Osama bin Laden, a US television reporter was working himself in a passion as he gestured to the compound behind him.

"It was one of the most secret places on earth," he enthused. "Now, it's a kind of tourist attraction."

In a way, he was correct. Around him were scores of other journalists and crowds of excited, jostling locals keen to make the most of the vantage point and the opportunity to catch at least a partial glimpse into the still-sealed compound. They may have been right to do so. The future of the house in which Bin Laden and his most trusted aides secretly lived remains unclear, but there are at least some people here who believe it should be torn down as quickly as possible.

If the US cannot allow Bin Laden to be buried in a grave through fear it may become a shrine, some reason, then it cannot afford to leave standing the house in which he lived.

"The government should destroy it. There are now not good memories associated with that building," said Yasir Khan, who owns a small tailor's shop just a few hundred yards from the compound. "We are happy that the biggest terrorist in the world has been killed here. This chapter is over. The world is now a safer place."

His views were echoed by Mohammad Rafique, an assistant superintendent of police, who said his town's reputation as a quiet, peaceful place had been forever destroyed.

"I'm very upset they were found here. We never used to have any extremists in the area," he said. "I didn't have a clue the house even existed. The house should be demolished. It would become a place of worship. The biggest tragedy is that it's such a peaceful town."

More details about the three-storey house and the people who lived there continue to emerge. Locals say the plot for the house was bought in 2005 by the thick-set Arshad Khan and his younger brother Tariq. Land registration documents obtained by the Associated Press reportedly show the buyer was named as Mohammed Arshad.

Whether any of these names were genuine is not clear. Reports have identified Arshad Khan as a trusted courier of Bin Laden who ultimately led the CIA and the US special forces to the compound, located in this town's Thanda Choha neighbourhood. Both were reported to have been killed in the raid in which Bin Laden was also shot dead.

Another trader whose shop is located close to the property said Arshad Khan and his brother dropped in once every week or so to buy biscuits and other basics. Mohammad Usman, whose shop sells vegetable oil in plastic bags and single sachets of shampoo and washing powder, said the two men often bought one or two cigarettes, though never a pack. Their preferred choice was always John Player Gold Leaf and it was always their habit to stand and enjoy their cigarettes while standing outside the shop.

"They never took a whole pack home with them," said Mr Usman. Could it be that Mr Bin Laden – the man who everybody here says they never actually saw – enforced a No Smoking policy at his home? As for the future of the property now, Mr Usman said: "This building is not the culprit. It should not be knocked down. It should be turned into a school for the children." The owner of a neighbouring shop, Rasheed Abbassi, saw more of the two brothers, and claimed they sometimes dropped by on a daily basis. From him they would purchase soft drinks, sweets for the half-dozen or more children who lived in the compound, eggs, salt and milk. They were always very polite, said Mr Abbassi, who added that he refused to believe Bin Laden had been the occupant of the property. "The US is just using this to attack Pakistan," he said.

Sardar Mehtab Ahmed Khan, a former provincial chief minister and a local MP, said many people in Abbottabad were anxious about what the coming months might bring. He said he did not believe Pakistan's ISI intelligence agency would be naive enough to knowingly allow Bin Laden to live in the town and said a regular flow of people from the country's tribal areas in the last five or so years might have accounted for no-one taking particular notice of the Bin Laden entourage moving in.

As for the property itself, he said it was true that Bin Laden "had been an inspiration to Muslims in Pakistan and the rest of the Muslim world", but did not believe the compound would become a place of pilgrimage for his supporters. "That is immaterial," he said.

Bin Laden's house, with its high walls and barbed wire and patches of black mould, was located inside the town's official military cantonment, an area in which properties are supposedly regulated by a series of rules, many of which the compound appeared to break. It is overseen by the Cantonment Board. Ultimately, however, the future of the house in which Bin Laden lived and died will by decided by the very highest levels of the Pakistani military. So far, there has been no official word on what will happen.

While that is the case, the tourists and the curious locals will continue to come and look.