The work aims to design a detection method to detect the defects on the rail surface quickly and accurately. Firstly, the gray-level equalization model was constructed by the gray-level mean of the image, and the gray-level value of the pixels in the rail surface image was corrected to overcome the influence of uneven illumination. Then, the spectral residual model of the image and phase spectrum were used to enhance the defects in the rail surface image. Secondly, threshold segmentation method was introduced to binarize the enhanced image, and the defective areas in the image were segmented and extracted accurately. Finally, the noise in the binary image was filtered by the filter, and the edge of the defect area and other details were preserved. Compared with the control group, the proposed method had better detection effect and higher precision and recall rate. The intuitive test results showed that the method proposed in this paper could detect the surface defects of the rail more completely. From the objective test results, the accuracy rate of the proposed method was 90.11% and the recall rate was 93.41%. At the same time, the objective test results also showed that the average time-consuming method was 45.17 ms. Compared with the control group method, the time-consuming was the least. Therefore, the method proposed in this paper could detect rail surface defects more accurately and rapidly. Therefore, the rail surface defect method proposed in this paper can not only accurately detect the rail surface defect, but also has high detection efficiency.