The work aims to remove noise in surface defect image of cold rolled aluminum sheet, keep resolution and details of the image, and avoid incorrect defect detection due to noise in the surface defect image of cold rolled aluminum sheet. Firstly, noise variance of the image was obtained by introducing a bilateral filtering method, and combining probability distribution function and the maximum likelihood function, and gray variance value in the bilateral filter function was adjusted to filter the noise in the surface defect image of cold rolled aluminum sheet. Then, in order to remove strong noise left after the bilateral filtering denoising, regional similarity model was constructed based upon the difference between pixels, and the strong noise in the image after denoising was determined by the bilateral filtering. Finally, the median filtering method was used to filter the strong noise while maintaining solution of the image, so as to further remove noise in the surface defect image of cold rolled aluminum sheet and keep the details and resolution of the images. In the method proposed herein, MSE value of the denoised images was 15.3743, 19.7713 and 23.7613, respectively, and PSNR value was 38.4971, 35.4792 and 31.1768, respectively when noise intensity was 0.09, 0.12 and 0.15, respectively. The proposed method can not only effectively remove the noise in the surface defect image of the cold rolled aluminum sheet, but also maintain image resolution and edge features satisfactorily, so that the denoised image has good visual effect.