The work aims to design an effective candidate window extraction method to improve the accuracy and real-time of surface defect detection of medium plates for problems of complex surface and low defect recognition rate in medium plates. The visual selective attention mechanism was introduced and generic object estimation algorithm based on Binarized Normed Gradients (BING) was used to quickly and accurately extract the defect region of interest (ROI) and shorten the search process effectively. Firstly, the samples were normalized to 8×8 size, and Normed Gradients (NG) were extracted to learn a linear SVM classifier for measuring saliency to predict the possibility of defects in image windows. Then, a linear SVM classifier calibrating the saliency score was learned by optimizing the saliency score at the sample scale. Finally, two SVM models were cascaded for on-line detection and extraction of defect interest region. The trained BING model and Inception-V3 convolutional networks were combined for defect detection and identification in medium plates. BING algorithm effectively reduced the number of windows of interest and achieved a recall rate of 98.2% when the number of ROI was 500. Under the premise of ensuring defect recall rate, the number of ROIs generated by BING is two orders of magnitude smaller than the sliding window method, which effectively reduces the computational complexity of subsequent identification modules and is conducive to introducing complex classifiers to improve the accuracy of surface defect detection of medium plates.