TOKYO, Japan (CNN) -- Japanese voters are at the polls Sunday in what is expected to be a tough contest for incumbent Prime Minister Junichiro Koizumi who is facing a public critical of his Iraq policy and pension reforms.

His Liberal Democratic Party needs to win 51 seats in the Upper House elections to consolidate power. If not, he faces a revolt from within his own party.

Voting started at 7 am (2200 GMT) and ends at 8 pm (1100 GMT).

Estimates put turnout at around 16 percent as of 11 a.m., down slightlly percentage from the same time in the 2001 upper house election, Reuters reported.

Koizumi was voted in 3 years ago on a wave of popularity with a whopping 80 percent approval rating.

Now he is struggling and his ratings have plunged to just 40 percent after he decided to keep Japanese troops in Iraq and pushed through an unpopular bill to reform the country's pension system.

The beleaguered system is unable to pay for its aging population and Koizumi's answer was to introduce legislation that increases payments and cuts payouts.

They were necessary reforms, Koizumi says, but it was not a popular policy.

On Iraq, the public is deeply divided over the wisdom of Koizumi's ambitious deployment -- Japan's riskiest mission since WWII.

When Koizumi announced that troops would be staying on after the Iraqi handover -- without consulting lawmakers -- the public was not pleased.

"The government is abusing its power. Since they represent the people of Japan, they should stand by us," voter Hiroko Furuya says.

The Japanese are clearly unhappy with Koizumi, but few are impressed with the opposition either. The result is that a chunk of former Koizumi supporters are now undecided.

"I wish the other parties would take a better position, but nobody seems capable," voter Takahiro Kaneshima.

"What he has to manage, and he's done a fairly good job so far, is manage his own party -- the conservative elements within his own party. And that is the true opposition to Koizumi," Asia Strategy's Keith Henry told CNN.

Prime Ministers have been forced to resign by their own party because of poor performance at the polls though analysts say that a resignation is unlikely this time.

But if Koizumi wins less than 51 seats, he will be on shaky political ground.