For the winter Olympics, NBC set out to measure how people across the U.S. are following its coverage—not just on TVs, but also tablets, smartphones and computers. Instead of using Nielsen Holdings NV, a powerhouse in TV ratings, the company turned to comScore Inc., better known for tracking digital media.

The tie-up is one sign of the intensifying competition facing Nielsen, whose sampling of viewing activity at 20,000 U.S....