Newton's first law, forces, charges

1. The problem statement, all variables and given/known data
Two uniform masses of .260kg are fixed at points A and B. Find the magnitude and direction of the initial acceleration of a uniform sphere with mass 0.01kg released from rest at point P that is acted on the gravitational forces of attraction from spheres A and B.

The diagram shows the spheres arranged in a triangle. If you were to divide the triangle in half, making two IDENTICAL right angles triangles, you would have a hypotenuse of 10cm, x value of 8cm, and y value of 6 cm.

2. Relevant equations
F=G Mm/r^2
G=6.67x10^-11
F=ma

3. The attempt at a solution

The x values cancel as vectors are equal in magnitude and opposite in direction.
This leaves the y values. Solve for F between mass of 0.26kg and 0.01kg.
F=(6.67x10^-11)(0.26)(0.01)/(0.06^2m)
=4.8x10^-11
Multiply this by two because there are two identical masses acting on the sphere at point P
=9.63x10^-11

Now, using F=ma, equate the result to this...
9.63x10^-11=ma9.63x10^-11=(0.01)a
divide 9.63x10^-11 by .01
a=9.0x10^-9

I understand that the two masses are not 0.06 m away. I am solving for the y component of distance between the 0.26mass and the 0.01 mass.

You are computing the force as if it was 0.06 m away and not taking components. You cannot get the components by inserting the component of the separation instead of the separation in the force law. You need to first compute the total force (using the actual distance) and then split it into components.

To see this more clearly, if things worked the way you are using the force law, the components would be larger than the total force, which is clearly not the case.