Don't believe me, believe the empirical academic research!
When economists finish their PhDs and are looking for a job they produce a job market paper. Given the incentives here, these tend to be particular impressive pieces of work, whether in method, data, estimation or topic, and this from Mario Alloza at UCL (his website) is no exception.

The paper (pdf) looks at changes in tax policy in the US and a representative sample of households between 1967 and 1996 and finds that a 1¢ in the $ rise in marginal tax rates leads to a 0.5%-1.3% decline in the probability of someone's changing income decile. For example, moving from being in the bottom 10% to being in the 10% who earn more than the bottom tenth, but less than everyone else.

According to Alloza's data, this effect is particularly significant at the bottom of the distribution—i.e. for the poorest people—and comes because taxes affect work incentives. When taxes are higher, that reduces the benefits to badly-off people from working harder and more and changing their prospects.

He says that we his result should make us more cautious of raising taxes to reduce inequality, because it will reduce opportunity and social mobility:

These empirical results have important implications for the design of fiscal policy. Tax reforms that reduce marginal rates are more likely to increase equality of opportunity (as measured by the degree of income mobility). This is because an attenuation of the distortionary effects of taxes in the labour market would make households more likely to take advantage of economic opportunities and move up in the income distribution. Therefore, fiscal policies that aim to reduce inequality should weight the trade-off in households’ welfare induced by the effect on income mobility.

Indeed, given all the costs of caring about inequality, and the very meagre benefits we gain from ameliorating it, perhaps we shouldn't care about it at all.