There is an urgent need for accurate measurement of pavement surface profile and smoothness index. This investigation compared the precision (repeatability) of the measured international roughness index (IRI) at the Virginia Smart Road for 19 profilers and their accuracy (bias) with respect to the rod-and-level reference measurement. Two existing accuracy criteria (ASTM Standard E950 and cross correlation) were compared for evaluating the relationship between the accuracy of profile elevation and IRI bias. It was found that there were profilers available that could produce the level of IRI accuracy and precision required for construction quality control and assurance. However, the accuracy and precision degraded when measuring hot-mix asphalt pavements with coarse texture and using the rod-and-level measurement as the reference. The cross-correlation method appeared to have some advantages with respect to the conventional elevation tolerance method for providing better consistency between profile accuracy and IRI bias. On the sites investigated, good cross correlation among the measured and reference profiles assured acceptable IRI accuracy.