WILLIAMSBURG, Virginia (CNN) --U.N. Secretary General Kofi Annan said Saturday that the issue of Iraqi disarmament was "not for any one state but for the international community as a whole" to resolve and urged the United States to seek consensus before attacking Iraq.

"When states decide to use force, not in self-defense but to deal with broader threats to international peace and security, there is no substitute for the unique legitimacy provided by the Security Council," Annan said in a speech marking the 310th anniversary of the College of William & Mary.

Annan added that the United Nations is successful when there is strong U.S. leadership "exercised through patient diplomatic persuasion and coalition-building. We all need to understand that the United Nations is not a separate or alien entity, seeking to impose its agenda on others. The United Nations is us; it is you and me."

Annan also noted, however, that it was the threat of U.S. military action that persuaded the Iraqi government to re-admit U.N. weapons inspectors. "Today, it is thanks in large part to the firm challenge issued by President Bush -- and the pressure that followed it -- that the inspectors are back in Iraq."

Annan warned Iraq that if it failed to make use of "this last chance, and continues its defiance, the council will have to make another grim choice, based on the findings of the inspectors ... When that time comes, the council must face up to its responsibilities."

If war results, "it may cause terrible loss and suffering to the Iraqi people and perhaps to their neighbors, too," Annan said. "We all -- first and foremost the leaders of Iraq itself -- have a duty to prevent this, if we possibly can."

"But our founders were not pacifists. They knew there would be times when force must be met by force."