ROME—Italian voters on Sunday rejected constitutional changes backed by the government, prompting Prime Minister Matteo Renzi to announce his resignation and handing populists a victory in the heartland of Europe.

With virtually all the votes counted early Monday, 59.6% were “no,” according to Italy’s Interior Ministry, marking a stinging rebuke to Mr. Renzi’s plan to overhaul Italy’s legislature to make it easier to pass laws,...