Nautical mile approximates an arcminute

Jan 1, 2003

To the editor: Larry McKenna's article, "Dealing with distortion," in the March/April 2001 issue, is an excellent introduction to map projections and the complications of applying them to the real earth. I was surprised, therefore, that the box accompanying the article on page 101 stated that "the nautical mile is defined as the distance on the earth's surface occupying one arc minute of latitude." Actually, the standard unit of measurement for navigators, the international nautical mile, is defined to be 1,852 meters (6,074.56 feet) exactly (see Bowditch, for example). The association of the nautical mile with an arcminute of latitude is only approximate.

The metric definition simply acknowledges the fact that, because the earth is not a perfect sphere, there cannot be a constant ratio between an angular and a linear measurement unit that holds true over its entire surface. For example, using the WGS-84 earth equatorial radius of 6,378,137 meters (20,920,289.36 feet), we find that the circumference of the earth at the equator is 21,688 nm by the metric definition. This gives the length of one arcminute of longitude at the equator as 1.00180 nm. The situation in latitude is complicated by the earth's 0.3 percent polar flattening (the actual value of the flattening is 1/298.257). Any geographic meridian is 21,602.5 nm.

However, the curvature of the earth's surface depends on the latitude, so that an arcminute of latitude varies from a minimum of 0.99509 nm at the equator to a maximum latitude of 1.00508 nm at the poles. Obviously, for practical purposes, these small dýpartures from the easily remembered "one nautical mile per arcminute" rule are unimportant. On the other hand, maintaining a distinction between length and angle is actually helpful in thinking about how the earth's shape affects our measurements of distance across its surface.

George H. Kaplan is an astronomer with the Astronomical Applications Department of the U.S. Naval Observatory in Washington, D.C.

[Editor's note: The commonly used value of pi at 3.14 is not accurate enough for these figures. Mr. Kaplan recommends using a value of pi to at least six or seven significant digits: 3.14159265.]