As far as I can tell, this problem does not have a simple solution. I have a horrible method for tackling it, which leads to a horrible answer. The idea is to use coordinate geometry on an equilateral triangle ABC with sides of unit length, to find a point P for which the distances PA, PB, PC are in the ratio PA:PB:PC = 4:5:6, and then to scale the result up so that these distances become 4, 5 and 6.

So let , and . The set of points for which the ratio of distances PB:PC is 5:6 is a circle with equation . The set of points for which the ratio of distances PB:PA is 5:4 is a circle with equation . Those circles intersect at the point given by

The distance PB is then . If we scale up the diagram to make PB = 5 then the side of the triangle becomes

If you do the calculation exactly, then the value for the side of the triangle comes out as Unless I have made arithmetical mistakes (something that unfortunately happens all too often), that expression cannot be simplified any further, which makes me think that this problem does not have a neat solution at all.