Objective Aiming at the problem of not having a unified evaluating criterion for the contrast test during the comparison with a foreign auto material lab on the surface roughness of cold-rolled steel sheets, systematic study was carried out on the correlation and consistency mentioned in the communication. Methods This paper elaborated the correlation coefficient test and the paired t-test both concerned in the selection of evaluation methodologies. Results By the contrast test data verification of the auto sheet surface roughness, the correlation coefficient test was only suitable for evaluating whether the two groups of test data was linearly correlated; the paired t-test was suitable to evaluate whether there was consistency between the averages of the two groups of test data. Based on application examples, a newly developed evaluation software of the contrast test was introduced. On the same interface, the software could not only display the quantitative evaluation conclusion of the paired t-test, but also visually display the evaluation result on the scattergram of the contrast test data via the value that was smaller than the limiting angle of ±1° standard of the angle formed by the line connecting the geometric center point to the origin of coordinate and the 45° ideal location line, and then draw a complete conclusion. ±1° standard was determined by a large number of statistical surface roughness contrast test data for the auto sheet. Conclusion Due to the uneven distribution of surface pits on the auto steel sheets, only the evaluation method for consistency could be used in the contrast test of auto sheet surface roughness.