Yes, it’s been a while. The sabbatical in Italy put a dent in testing, sure. Then came a disc bulge which took any decent running load out of the picture. On the upside I took the time out to brush up on my geospatial-data-dorking skills in R and PostGIS and rebuilt the process entirely in R script. Yes, it’s nerd. The result is a complete re-analysis of collected data to give better insight into the capabilities of the GPS models tested.

In short, the updated methodology is leaning more towards quantified GPS performance bench-marking rather than a subjective review. Plus the future collection and analysis workflow is super easy which equal less typing and more running. Or beer, or whatever.

We can make pretty pictures like this track point cloud that shows GPS positional accuracy over a large number of runs (hint – this is a recent model GPS).

Having powered up the analysis we can now see the impact of tree-cover, bendy paths, running speed, sampling rate, satellite availability, and even how much the satellite data is actually filtered out from the recorded distance (ie. does the watch work like a glorified activity tracker). The results give some great insights on which watches work best in actual trail conditions. With a couple of statistical tests we can formally identify which comes out as the better, or worse, performers. And the data can be modeled to show which factors effect watch accuracy.

As a teaser, the chart below shows the accuracy of all models across easy, mixed, and difficult conditions as recorded by the watch (ie. as you see it) in purple and as recorded by the raw GPX data (ie. buried underneath what you see) in yellow. The brown is the overlap of the two. The dashed lines are means, and grey is the true distance. We can see here the accuracy deteriorating between the conditions nicely. At the individual watch level there are huge difference in performance between these categories. Given we normally spend a lot of time running in the mixed to difficult conditions (when we are not testing) this analysis gives a great view on which watch will give us the good trail running numbers.

Plus, we’ve sold a kidney and promised a first born child to get hold of a Suunto Spartan Ultra for testing. The first round of formal surveyed trail testing is done, though sadly the +20hr endurance run test is off the cards for now (back prognosis points to a Spring running recovery). At the same time we tested a Sony Xperia Z5c using the SportTracker app. The results are interesting… our advice would be hold off any purchase till you read the review.

We’ll be writing up the Sunnto Spartan Ultra and Sony Xperia Z5c and updating existing reviews over the coming weeks. And data collection is in progress for the TomTom Adventurer.

Ok, after a year and a bit we’ve finally exhausted the MEC’s pool of GPS trail running watches. This has included Garmin’s FR310XT, FR910XT, fēnix 2, and fēnix 3, Suunto’s Ambit2, and Ambit3 Peak, and Polar’s V800. Having put all these through extensive and objective real world trail testing we learnt a few things.

How to test

The context of testing is critical, just because it performs in a suburban park doesn’t mean much in the trail conditions you are likely to encounter in NZ. And sample size matters, some issues just don’t present after a weekend’s worth of fun. You need to repeat the test again, and again. And again. All the time controlling for conditions. Finally you need to test against some actual known truth, ie. a properly surveyed course.

Who to believe

Watch marketing is obviously heavily pitched towards the feature list and the vast majority of review sites do little more than extend the marketing reach of the brands. ‘Good’ reviews close to release date may shift product, but are ultimately meaningless if they don’t test what is important to you. And explaining features is not really testing, it’s like counting the tyres and calling it a race car. So ask yourself, honestly, is your review site of choice really testing or just teasing with a bunch of nice pics and gadget jargon.

We get it, it’s difficult and time consuming undertaking quantitative real world reviews with so many factors in play. But we care a lot about accuracy. We also are rather keen on reliability and the practical implementation of features (aka functionality). And true battery life in real trail running situations, we really do absolutely need to know if a watch can last more than 12hrs. Bottom line for us, performance equates to authenticity at this price point.

And you know what, it turns out you can’t take accuracy, performance, or reliability for granted. Even models from the same brand can fare very differently. We are reviewing these GPS watches simply because we want to see great trail watches from a variety of brands. That’s it. No sales agenda, no brand preference.

And the envelope please…

So our pick of trail GPS watches for NZ conditions? If accuracy, reliability, and endurance are your thing Suunto’s Ambit3 Peak is well clear of the rest. It tops all the accuracy tests we measure, and won the battery endurance test to boot. It’s not all roses though, the screen is a bit limited, there’s no vibrate alert, buttons are rather non-responsive, course navigation is basic (though still beats the competition with waypoint alerts on the breadcrumb course), and the battery savings modes aren’t very ultra friendly, and that charging clip! Also the Android mobile app is still not up to standard with daily re-pairing required (July 7 update – Ambit3 FW version 2.2.16 with app version 1.3.1 has survived a couple of days without losing connection).

If you don’t want to splash out big bucks and still value accuracy and endurance if you can find a Suunto Ambit2 or Garmin FR910XT you’d be well served (note the 910XT not the 920XT – which we haven’t tested).

And if you absolutely must have all the gazzilion features of the fēnix 3, or simply wish to support Garmin for their excellent openness towards data, you’ll be ceding a fair amount of accuracy and endurance (without a battery I.V. hookup). That said the fēnix 3’s screen, vibrate, and positive feel buttons make the user experience better than the rest.

So what now?

So now we’ve finished with the pool of available watches now what? Sure there are some more recent models, Garmin’s FT920XT, the fēnix 3 HR, and FR735XT. Yet I can’t see anyone in MEC ditching their current watches as none of those really add anything to the trail, and all have reduced battery life. Suunto’s pending Spartan Ultra looks intriguing, though with the touch screen and lack of detailed specs we’ll be waiting to see how that pans out before leaping in (especially at that price point).

In the interim we are reviewing Sony’s waterproof Xperia Z5 Compact with the Sports Tracker app (another Finn) as the combo gives barometric altitude and manual lapping capabilities. At present it’s beating about half the GPS watches in terms of accuracy. Looking forward to seeing if we can outrun it in the battery endurance test.

Also our lead tester, Ron, is about to go on a 6 month sabbatical in Italy so lots of running but not so much testing.