The general form of the relevant equations is
\begin{align*}
m &= \iint \rho\,dx\,dy \\
M_x &= \iint \rho y\,dx\,dy \\
M_y &= \iint \rho x\,dx\,dy
\end{align*}
When the region of interest is between x=a and x=b and is bounded on the top by f(x) and on the bottom by g(x), you get the equations you cited. For instance, for the moment about the x-axis, you get
$$M_x = \rho \int_a^b \int_{g(x)}^{f(x)} y\,dy\,dx = \rho \int_a^b \left.\frac{y^2}{2}\right|_{g(x)}^{f(x)} \,dx = \frac{1}{2}\rho \int_a^b [f(x)^2-g(x)^2]\,dx$$
If you sketch the region for this particular problem, however, you'll see the roles of x and y appear to be reversed, so the formulas you were trying to use don't work. You'll need to derive the correct ones or adapt the ones you have for this particular case.