The work aims to propose a new algorithm in surface defect detection of metal workpiece based on an improved RCNN method in view of the limitation of traditional detection algorithm in the detection of workpiece surface defects, as well as the problems of low precision, low accuracy and tedious detection process. In the process of image pre-processing, the methods of image defect location and annotation and image data enhancement were used. While in model training, in order to avoid the shortage of some classification data and avoid the over fitting phenomenon of system test model caused by too small data set, the original image was processed through data amplification. In the design of detection network model, the non-maximum suppression algorithm was used to filter candidate regions of defect image so that a regional suggestion network was constructed, which realized the reuse and fusion of multi-layer network features and improved the detection precision of the system on the basis of reducing the redundancy of candidate. A multi-level ROI pool layer structure design algorithm was introduced to eliminate the system deviation caused by ROI pooling and rounding, which could effectively and accurately detect the surface defects of parts. The position coordinate of original drawing based on ROI-Align algorithm was improved and the position coordinate of original drawing was obtained by bilinear interpolation method, which overcame the problem of pixel position deviation and detection misalignment caused by ROI-Pooling design algorithm based on nearest neighbor interpolation method. The detection method proposed in this paper proved that in the test set, the detection speed of the target defects on the surface of metal workpiece was 22 fps, the accuracy rate was 97.36%, and the recall rate was 95.62%. Compared with the traditional workpiece surface detection method, the improved Faster RCNN method has faster speed and higher accuracy for target identification and positioning processing, which can improve the detection performance of workpiece surface defects under complex environment.