Surface roughness has a significant impact on the wear resistance, corrosion resistance, and reliability of components. Accurate prediction of surface roughness can effectively control the manufacturing process and optimize processing parameters. However, the coupling effects of various influencing factors on surface roughness obscure the formation mechanism of profiles, leading to technical challenges in the insufficient predictive accuracy of mathematical models for surface roughness intelligent control in industrial applications. This study established a mathematical model for surface roughness in face milling to address this issue. Firstly, the surface profile forming mechanism of face milling workpiece was analyzed and the surface profile model along the tool feed direction was established based on tool geometry and machining kinematics, taking into consideration the tool runout boundary conditions. The mapping between dynamic factors (tool wear, tool vibration, elastic recovery) and surface roughness was established through the milling force. The compensation function of profile height deviation about milling force was established and resolved by a convolutional neural network (CNN) which contained 5 convolutional layers and 3 fully connected layers. Next, the mathematical model for surface roughness in face milling was developed, with Ra serving as a characterization parameter for surface roughness. Finally, the face milling ZG32MnMo experiment was carried out to collect the profile data and milling force signals respectively, which used indexable face milling cutters. The milling data set was established with cutting force as input and profile height deviation data as output. The CNN was trained and the profile height compensation values was analyzed. CNN training results showed RMSE of 0.81 μm and 0.84 μm for the training and test sets, respectively. Through CNN, compensation values for profile height were analyzed to enhance the prediction accuracy of the mathematical model for surface roughness. The surface roughness mathematical model accuracy was validated. Surface roughness mathematical model considering tool runout and surface roughness mathematical model considering tool runout optimized by CNN were compared in terms of accuracy. The results showed that the surface roughness mathematical model considering tool runout optimized by CNN and the surface roughness mathematical model considering tool runout predicted Rsm and Rmr(c) close to the experimental values in the overlapping and non-overlapping along the feed direction, and the surface roughness mathematical model considering tool runout optimized by CNN exhibited higher predictive accuracy for Rmr(c) compared with the surface roughness mathematical model considering tool runout. The surface roughness mathematical model considering tool runout optimized by CNN predicted errors for Ra in the overlapping and non-overlapping along the tool feed direction to be 18.71% and 14.14%, respectively, while the surface roughness mathematical model considering tool runout predicted errors of 29.32% and 46.97% in the same regions. The surface roughness mathematical model considering tool runout optimized by CNN improved the accuracy by 10.61% and 32.83%, respectively, compared with the surface roughness mathematical model considering tool runout. The surface roughness mathematical model established by considering the boundary conditions of tool runout and dynamic milling force coupling can effectively characterize the surface roughness of face milling and provide reference for its application in quality control engineering.