There are a number of ways to do this. Most of them involve some basic linear algebra.

Since the shape is convex, you have more options available to you. The way I'd do it is to form lines connecting the four corners of that rectangle. Make sure the lines are all oriented so they go clockwise:

Lower Left - Upper Left
Upper Left - Upper Right
Upper Right - Lower Right
Lower Right - Lower Left

You can then use the perpendicular dot product to determine whether or not a point is to the "right" or to the "left" of a given line.

Since all lines are oriented clockwise, in order for the point to be inside the shape... it must be to the "right" of all 4 lines. Therefore you just check the point against all 4 lines and if it is to the left of any of them, you know it is outside the area.

Perpendicular dot product is pretty simple. Assuming you have a 'Vector' or 'Point' class (that has x,y components):

Another option (if you don't have a convex shape) is to form a ray from the target point, and count how many sides of the shape that ray intersects. Even number = point is outside the shape, odd number = point is inside the shape.

The math to do this efficiently is a little more involved so I won't get into it here. The above solution is easier.

what i have is a image i use as a button but i would like to detect if mouse click is in part of the image. its not a normal shape.

so i thought is there away to get the pos of the mouse click on the displayed image an then compare that with a image in resource thats a image mask.
so that it checks if the mouse click clicked at position of a white pixel?

Depending on what you're doing... that might be good enough. The user usually won't be able to tell whether or not collision detection is pixel-perfect, so you can often just create a box that approximates the area of the object.

But again it depends. I'm not really sure what it is you're going for, I'm just throwing out ideas.