"To link Sardar Patel with any party will be a big injustice to him (Patel). That party (Congress) was a part of his life and both history and Modi cannot deny it," the BJP's prime ministerial candidate said.

Modi spoke at an event after Bharatiya Janata Party leader L.K. Advani laid the foundation stone of a huge proposed statue of Sardar Patel here.

Hitting out at Congress accusations that he (Modi) was trying to usurp Sardar Patel's legacy, Modi said leaders such as Rana Pratap, Bhagat Singh and Shivaji were not from the BJP but he respected them all.

Modi said though Mahatma Gandhi fought to free the country of untouchability, "political untouchability" still existed. "We should not also try and remove political untouchability."

After Modi said Tuesday that he wished Sardar Patel had been India's prime minister, Manmohan Singh responded that the "Iron Man" had always been a Congressman and secular too.

"I appreciate his remarks that Sardar Patel was truly secular. We also say that this country needs secularism like Sardar Patel's and not the vote bank type," Modi said.

"He (Sardar Patel) was secular but that didn't come in the way of his rebuilding the Somanth Temple."

The proposed statue of Sardar Patel will be the tallest statue in the world, Modi has said. It will be double the size of the Statue of Liberty.