In the duel over economic records between President Barack Obama and Republican challenger Mitt Romney, a Romney surrogate this week took the fight straight to our wallets.

When Romney was governor, adviser Ed Gillespie said on Fox News Sunday, "the average income for a family in Massachusetts went up by $5,500."

High unemployment and stagnant wages are hurting Americans everywhere, so we decided to check whether Massachusetts families fared that much better during Romney’s 2003-07 gubernatorial term.

First, the data

The Romney campaign cited U.S. Census data when we asked for backup for Gillespie’s statement. Spokesman Ryan Williams pointed us to Massachusetts’ median household incomes from 2002, the last year before Romney became governor, through 2006, the final full year he was in office. Here are the figures:

2002: $49,855
2003: $50,955
2004: $52,019
2005: $56,017
2006: $55,330

So the hard numbers are accurate. The median income in Massachusetts rose $5,475 from 2002 to 2006.

Romney’s influence

When rating claims such as these, PolitiFact weighs not only whether the claim is numerically correct but also whether it’s appropriate for the speaker to assign credit or blame.

Said Fred Bayles, a former reporter who covered Romney’s administration: "Any set of numbers related to the economy is kind of like reporting on the tides and ascribing an individual responsibility to a force of nature."

Barry Bluestone, dean of the school of Public Policy and Urban Affairs at Northeastern University, said that "governors, in general, have little direct impact on employment or income in the short run. They can set the stage for future growth in jobs and income if they invest heavily in education and in infrastructure, but this did not occur during Romney’s term in office."

He also noted that the census income figures are not adjusted for inflation, a point echoed by Andrew Bagley, director of research and public affairs for the Massachusetts Taxpayers Foundation.

Bagley crunched the numbers for us taking inflation into account. Using data from Moody’s Analytics, the median household income in Massachusetts increased by roughly 2.8 percent annual growth during Romney’s term, Bagley said. The nationwide consumer price index (CPI), which shows how much inflation grew during the same period, was 2.9 percent per year.

"So, the answer provided by Gillespie appears accurate. But, if someone said that ‘real’ Massachusetts median household income fell during Gov. Romney’s four years in office from 2002 Q4 to 2006 Q4, that would also be accurate because the average annual growth of 2.8 percent was below the inflation rate of 2.9 percent, so real incomes actually declined," Bagley wrote in an email.

When you factor that in with slow job growth -- Massachusetts private-sector employment grew by just 0.79 percent during Romney’s term -- Bluestone said the outcome is not surprising.

"There really was nothing that Romney accomplished as governor that would have increased jobs or income," he said. Bluestone has donated $700 to Obama's campaign since 2011.

Bayles, who now directs the Boston University Statehouse Program, also added that Massachusetts’ income gains lagged behind the national average, 11 percent compared with 14.2 percent.

"Does that mean Romney could be criticized for Massachusetts underperforming the national average? Probably not."

He said outside economic forces made the biggest difference.

"The Massachusetts economy took a slightly bigger hit when the tech bubble burst and was a bit slower in recovering. There were forces much larger than a four-year term that impacted all these numbers," Bayles said.

Jeffrey Berry, a political scientist at Tufts University, summed it up this way: "The macroeconomic changes instituted by the Romney administration were far too modest to be responsible for a change in average income of this magnitude. It's far more likely that most of the change in Massachusetts were a reflection of national policies and trends."

Our ruling

Gillespie’s statement that median family incomes in Massachusetts grew by $5,500 during Romney’s term is numerically accurate.

However, that dollar figure is not adjusted for inflation, and two experts told us that after inflation is taken into account, real incomes in Massachusetts actually declined slightly. What’s more, governors in general have little impact on immediate-term gains in jobs and incomes.

Gillespie’s statement leaves out those important details, leaving the inaccurate impression that Massachusetts families were significantly better off under Romney. We rate it Half True.

Update: This article has been updated to include Barry Bluestone's political contribution.