The level curve for the utility function $ \ R \ $ for $ \ U \ = \ 0 \ $ is the blue ellipse, with the lines of its horizontal and vertical axes shown; the constraint parabola is drawn in red, with its symmetry axes and line of latus rectum marked. The "close-up" graph at right shows the curves where they come close to meeting.

We find that the chosen level curve comes quite close to contacting the constraint parabola. From the above equation for the ellipses, we see that smaller values of $ \ U \ $ would make the ellipse intersect the parabola, while larger values would cause it to miss the constraint entirely. So we are in fact in the process of determining the maximal value for $ \ U \ $ , and that said value will be something close to zero.

We can confirm that estimate by noting that the tangent point between the appropriate level curve of the utility function and the constraint curve falls in the vicinity of that upper endpoint of the latus rectum . The distance from the center of the elliptical level curves to that point is

This is pretty close to the length of the "semi-vertical axis" of the level ellipse for $ \ U \ = \ 0 \ $ , which is $ \ b \ = \ \sqrt{\frac{8162}{12}} \ \approx \ 26.08 \ $ . So we would expect the maximal value for $ \ U \ $ not to be far from zero.

We could, of course, simply use a graphing device to experiment with various values of $ \ U \ $ to see the approximate value for which a point of tangency can be produced. We find that $ \ U \ = \ -1 \ $ doesn't quite reach the constraint parabola [in red], while $ \ U \ = \ -2 \ $ crosses the parabola at two points. So we have a candidate interval of $ \ -2 \ < \ U \ < -1 \ $ for the maximal value of the utility function.

We would like to improve on this estimate by applying Lagrange multipliers. As you discovered, however, an algebraic solution alone is not very helpful. We can conclude that

is the (rotated) hyperbola on which the tangent point between the maximal utility level curve and the constraint parabola lies (the lines $ \ x \ + \ y \ = \ 0 \ $ and $ \ x \ + \ y \ = \ 2 \ $ are excluded). Unfortunately, combining this last equation with that for either the parabola or the ellipses doesn't present a convenient algebraic solution. Instead, we will just graph the "Lagrange" hyperbola [in blue] and the constraint parabola [in red]; the excluded lines are in black:

The tangent point is found to lie very nearly at $ \ (x, \ y) \ = \ (4.25, \ -1.60) \ $ . Upon inserting these coordinates into the level ellipse equation, we obtain

It should be noted that our result is rather sensitive to "round-off errors", but we can reasonably conclude that the maximum value for the utility function under the given constraint is $ \ R(4.25, \ -1.60) \ \sim \ -1 \ $ .