David Cameron has defended his decision to stop short of delivering a formal British apology for the Amritsar massacre in 1919, in which at least 379 innocent Indians were killed.

As relatives of the victims expressed disappointment, the prime minister said it would be wrong to "reach back into history" and apologise for the wrongs of British colonialism.

He was speaking shortly after becoming the first serving British prime minister to visit the scene of the massacre, which emboldened the Indian independence movement. He bowed his head at the memorial, in the Jallianwala Bagh public gardens. In a handwritten note in the book of condolence for victims of the massacre, Cameron quoted Winston Churchill's remarks from 1920. He described the shootings, in his own words, as a "deeply shameful event".

As he prepared to leave Amritsar, Cameron explained why he had decided against issuing an apology. "In my view," he said, "we are dealing with something here that happened a good 40 years before I was even born, and which Winston Churchill described as 'monstrous' at the time and the British government rightly condemned at the time. So I don't think the right thing is to reach back into history and to seek out things you can apologise for.

"I think the right thing is to acknowledge what happened, to recall what happened, to show respect and understanding for what happened.

"That is why the words I used are right: to pay respect to those who lost their lives, to remember what happened, to learn the lessons, to reflect on the fact that those who were responsible were rightly criticised at the time, to learn from the bad and to cherish the good."

Among the relatives of the victims who were disappointed that the prime minister had not apologised was Sunil Kapoor, whose great grandfather Waso Mal Kapoor died in the shootings. He said: "If he said it is shameful, why did he not apologise?"

Kapoor, president of the Jallianwala Bagh Freedom Fighters' Foundation, said: "I am not satisfied that he did not meet the families. We have waited 94 years for justice."

Sunil Kapoor's great-grandfather was among the dead at Amritsar Nicholas Watt

Cameron said Britain could still be proud of its former empire – while acknowledging the mistakes – as he rejected demands to return the Koh-i-Noor diamond to India from the British crown jewels.

He said: "I think there is an enormous amount to be proud of in what the British empire did and was responsible for. But of course there were bad events as well as good events. The bad events we should learn from and the good events we should celebrate.

"In terms of our relationship with India is our past a help or a handicap? I would say, net-net, it is a help, because of the shared history, culture, and the things we share and the contributions that Indians talk about that we have made."

Asked whether Britain should return the diamond, he said:

"I don't think that is the right approach. It is the same question with the Elgin marbles," he said. "It is for the British Museum and other cultural centres to do exactly what they do do, which is link up with museums all over the world to make sure that the things we have, and are looked after so well, are properly shared with people around the world. No, I certainly don't believe in returnism."

The Indians were shot dead in Amritsar by riflemen acting on the orders of Brigadier General Reginald Dyer. No 10 believes there is no need to apologise because the British state condemned Dyer's actions at the time. As war secretary in 1920, Churchill described the shootings as "a monstrous event, an event which stands in singular and sinister isolation".

Sukumar Mukhajee, secretary of the memorial committee, whose grandfather survived the shootings, welcomed Cameron's remarks. Mukhajee, who met the prime minister, said: "He has come here. He has paid his tribute. It is more than an apology."

Anita Anand, the BBC presenter, tweeted during Cameron's visit: "My grandfather was one of the lucky few who survived."

The prime minister hopes his strong condemnation of the shootings will help Britain and India to move on from what the Queen has described as the sadness of the past. He believes he is on firm ground in declining to apologise because of Churchill's strong language a few months after Dyer was forced to retire.

Churchill told the House of Commons on 8 July 1920: "That is an episode which appears to me to be without precedent or parallel in the modern history of the British empire. It is an event of an entirely different order from any of those tragical occurrences which take place when troops are brought into collision with the civil population."

The Golden Temple, the holiest site in the Sikh religion, receives at least 100,000 visitors a day. Photograph: Stefan Rousseau/PA

The prime minister, who has an eye on the Sikh vote in Britain, paid an hour-long visit to the Golden Temple in Amritsar. Out of respect to Sikhs as he visited their holiest site, he wore a dark blue bandana on his head.

The prime minister said he had been moved by his visit to the Golden Temple. "Today was fascinating and illuminating – to go to the place that is so central to the Sikh religion. I am proud to be the first British prime minister to go and visit the Golden Temple and see what an extraordinary place it is – very moving, very serene, very spiritual. It was a huge honour and a great thing to be able to do. I learnt a lot.

"In coming here, to Amritsar, we should also celebrate the immense contribution that people from the Punjab play in Britain – the role they play, what they give to our country. What they contribute to our country is outstanding.

"It is important to understand that, to pay respect to that, and to seek a greater understanding of the Sikh religion. And that is why the visit to the holy temple, the Golden Temple, was so important."