and when i check error's with glError after gluUnProjectit give's invalid operation.

and when i put DebugGL to canvas it complain's about every gl.glEnd();and when i remove them it work's but when i click on canvasso that it call's to method that contains gluUnproject it give's me out of memory exception.

now i know the problem,it isn't gluUnProject that is failing,it's glGetDoublev and glGetIntegerv.those return's full of zero's arrayfor GL.GL_MODELVIEW_MATRIX,GL.GL_PROJECTION_MATRIX and GL.GL_VIEWPORT.

I tried the glGet calls exactly as you had them written, and they worked fine - then I noticed you said this. Odds are you're calling glGet from a different thread that the GL context is currently active in. There is a way to change this (i noticed it in the Animator source) but you'll probably have nasty race conditions if you start tinkering with that.

My advice would be to set a flag to indicate that your GL thread should project the point, and possibly do the work there, then you avoid most of these issues.

java-gaming.org is not responsible for the content posted by its members, including references to external websites,
and other references that may or may not have a relation with our primarily
gaming and game production oriented community.
inquiries and complaints can be sent via email to the info‑account of the
company managing the website of java‑gaming.org