I am trying to make a custom ToolButton but I am having some problems getting correct measurements.

I have drawn a line (using the normal line tool) and set its length to 1000m. (Double-clicking it and measuring it with the ruler tool both show it to be exactly 1000.00m)

However when I use my tool and drag from one end of the line to the other it gives a distance of 999.8892m. I'm guessing it's some kind of projection issue but I'm using the same settings throughout the system... Where am I going wrong?

If your tool involves interacting with visible points on the screen with a mouse, please remember that the mouse is internally giving coordinates to the nearest pixel. On a high-resolution screen there will be 0.3 to 1.0 meters per pixel. Anything less is just measurement error. When measuring the length of a line segment with a mouse, there will be two independent errors of this amount at the two endpoints, resulting in a magnification of the imprecision by sqrt(2). This leads to 0.5 to 1.4 meters imprecision in the total length. In particular, 1000m = 999.8892m at this level of precision.
– whuber♦Oct 24 '11 at 17:07

I don't know, but I believe a good snapping routine would snap to vertex coordinates rather than pixel coordinates on the screen. But length measurements on screens are intended to be crude: they're one-off checks, not high-precision calculations. Therefore software developers would feel free to use quick approximate formulas and they might also use a different projection than you expect. The question comes down to what your "tool" is really intended for.
– whuber♦Oct 26 '11 at 14:43

Thanks Peter! I'm still a little confused though. It was returning -1 (meaning it's a long/lat coord system which is fair enough). However switching to cartesian in all the menus I could find made no difference. I got it to work by doing a "set coordsys" statement using the coordsys clause for British national Grid. But this is the projection that all my layers and system should be using anyway. What did "set coordsys" do that the menu system didn't? Thanks again.
– Mr_ChimpOct 24 '11 at 13:26

1

@Mr_Chimp I'm pretty sure that MapBasic apps can have their own coordsys that isn't the same as the systems, which is why you have to tell it what to use. I haven't done MapBasic in a long time so I might be wrong.
– Nathan WOct 25 '11 at 1:38

1

Yes, the coordinate system used by a MapBasic application is independent of that used in a map window. To ensure consistency, you should replace the Set Coordsys Earth line with set coordsys window frontwindow(). This will force MapBasic to use the coordinate system in use in the active window, in this case the map window that the tool was used on. An inappropriate coordinate system will affect whether the cartesian or spherical technique is used by the Distance function, and hence may be the cause of the discrepancy.
– Andy HarfootOct 25 '11 at 16:24

@Nathan W - That would make a whole bunch of sense. Thanks!
– Mr_ChimpOct 25 '11 at 16:25

As Peter says, MapInfo has two different approaches to distance measurements - cartesian, or spherical, the latter taking into account the curvature of the earth. The approach used can be set for individual map windows by altering the settings found under Map > Options, the default setting for new map windows can be set under Options > Settings > Map Window

Check which setting is being used when drawing and measuring the line in the Map window - you should see the words Spherical or Cartesian appearing after the measurement. Check whether this corresponds to the method being used by your MapBasic program by choosing the appropriate function from those described by Peter.

Giving the distance in metres gives a similarly incorrect result. I changed it to cm in order to test it. Using the ruler when measuring in cm gives a length of 100000 cm which is correct. Using my tool in cm gives 99988.92. I could round the answer but that is just avoiding the problem.
– Mr_ChimpOct 24 '11 at 11:32