The work aims to provide an effective method for detecting surface defects of metal products, so as to monitor surface quality of metal products. Firstly, adaptive median filtering method was introduced to filter noise in original image, so as to improve detection accuracy of metal surface defects. Then, reciprocal of image gradient was used to improve diffusion factor of traditional P-M nonlinear diffusion model, so that such areas with higher gradient value in the metal surface image could be smooth, while smoothness of other areas remained unchanged. The difference between the original image of metal surface and the image after nonlinear diffusion was applied to eliminate the influence of illumination on metal surface image, and obtain the image of metal surface with uniform background, so as to enhance the contrast between defect area and non-defect area. Finally, the adaptive two valued model was constructed based upon standard deviation of image block in the differential image, the differential image was binarized to extract defect area on metal surface and detect metal surface defects accurately. Detection of cracks, scratches, notch and rust defect images showed that this method could be used to accurately detect metal surface defects. The method designed in this paper can detect metal surface defects, and detection accuracy is also superior to other detection methods of metal surface defects.