Although a linear least squares fit of a circle to 2D data can be computed, this is not the solution which minimizes the distances from the points to the fitted circle (geometric error). The linear solution minimizes the algebraic error of a function something like
f(x) = ax'x + b'x + c = 0

Minising the geometric error is a nonlinear least squares problem. fitcircle allows you to compute either - it uses the algebraic fit as the initial guess for the geometric error minimization.

Also, @Graeme, the accuracy of the fitted circle depends on the kinds of errors that are in your data. If you assume there is a true underlying set of parameters that you're trying to find, and that your data is normally distributed, then the accuracy will decrease with number of data points as something like s / sqrt(n), where s is the standard deviation of perpendicular distances to the fitted circle, and n the number of data points. The error will then probably follow some kind of t distribution.

This code computes a least squares fit, which is not robust to large outliers - the results you got are not unexpected, nor are they wrong.

Because the objective being minimised is a function of the square of the distance, big outliers will skew the fit. If your data is known to have significant outliers, least squares is not a good choice of objective to minimise.