2 Answers
2

If you understand $A,B,C,D,x$ as complex numbers then your condition is
$$\frac{x-A}{x-B}/\frac{x-C}{x-D}\in\mathbb{R}.$$
Let us denote that real number $t$, i.e. you have equation
$$(x-A)(x-D)=t(x-B)(x-C).$$
For any given $t$ it is a quadratic equation for $x$, so we can solve it; the solution doesn't look very pretty:
$$x = \frac{\pm\sqrt{(-A+B t+C t-D)^2-4 (1-t) (A D-B C t)}-A+B t+C t-D}{2 (t-1)}.$$
Anyway, this gives you the points you're looking for (parametrized by $t\in\mathbb{R}$).

For a "formula" we would first have to discuss what constitutes an answer, but I made a picture to make it clear that the condition "inside" is not a very natural one.

I used geogebra. Note that when the curve crosses the line CD or AB , you do not have equal angles anymore, instead the smaller angles sum to 180 degrees, but it is fine again when the curve crosses again.

Furthermore, note that if you move the vertex A slightly, the part of the curve that passes through $A$ and $B$ becomes detached and formes a little oval curve.

@lhf: Well, yes, so the point is that it is not too hard to find a polynomial equation for the curve (using coordinates and equality of cosines expressed with inner products), but the degree would be quite high and the question of being inside even more intractable than the question of an explicit formula.
–
PhiraMay 10 '11 at 17:01

My visualization skills are failing me now, but it looks as if the "arrowhead" case might be an even more perverse configuration...
–
Guess who it is.May 10 '11 at 18:08

1

@I.J. Kennedy: Geogebra has a function that draws the locus of points when a certain point moves along a line. In my first picture you can see the point $E$ that moves along the bisector of $AB$. $E$ was the center of a circle through $A$ and then I drew a circle with corresponding angles through $CD$, then I defined the intersection of the two circles as $I,J$ and let geogebra draw the locus of $I,J$ when $E$ moves along the bisector.
–
PhiraMay 10 '11 at 20:43

1

@user3749 I want to point out that you have voted neither for my answer nor for the other answer yet. You said you would be looking up the connection of cosinus and inner product and this is exactly what I used. Angles equal implies cosinus equal. And in any case, I have to go to bed.
–
PhiraMay 10 '11 at 23:45