Compared with flat city metro, mountain city metro has the characteristics of complex lines with large climbing slopes and many curves. Its rail corrugation forms are more diverse. This paper is aimed at exploring the generation mechanism of rail corrugation in braking section of mountain city metro, and proposing relevant control method of rail corrugation. Based on the theory that rail corrugation is induced by friction self-excited vibration, the rail corrugation in braking section of mountain city metro is studied. Firstly, according to the field investigation of Chongqing Rail Transit Line 10, it is found that the rail corrugation with wavelength of 40-50 mm occurs frequently in braking section of mountain city metro. After that, combined with the field investigation, the contact model of the wheelset-track-brake system is obtained. According to the contact model of the wheelset-track-brake system, the finite element model of the wheelset-track-brake system is built. Then, the friction self-excited vibration characteristics of the wheelset-track-brake system are studied by complex eigenvalue analysis method. Furthermore, the influence of fastener parameters and surface texture of brake pad on the friction self-excited vibration characteristics of the wheelset-track-brake system is studied. Two fitting equations are applied to fit the fastener parameters by least square method. The root mean square error of fitting equation I is 3.99, and the root mean square error of fitting equation II is 4.39. Based on the two fitting equations, the optimal solution of fastener parameters is obtained by genetic algorithm and particle swarm optimization algorithm. The results show that there are two unstable vibration frequencies of 653.73 Hz and 584.76 Hz in the wheel set-track-brake system. Under the corresponding modes, both rails occur unstable vibration. The real part of the complex eigenvalue of 653.73 Hz is larger, which is 46.53. Comparing surface texture and no surface texture of brake pad, it is found that the real part of the complex eigenvalue of the wheelset-track-brake system with surface texture of brake pad is smaller. Comparing different surface textures of brake pad, the real part of the complex eigenvalue of the wheelset-track-brake system with regular hexagonal texture of brake pad is the smallest, which is 19.46. Comparing four fastener parameters, it is found that vertical stiffness and lateral stiffness of fastener parameters have a great influence on the friction self-excited vibration characteristics of the wheelset-track-brake system. Comparing fastener parameters before and after optimization, the real part of the complex eigenvalue of the wheelset-track-brake system with the optimal solution of fastener parameters is smaller, which is 0.49. In conclusion, the friction between wheelset-rail subsystem and brake subsystem leads to the friction self-excited vibration of the wheelset-track-brake system. And it is an important factor to induce rail corrugation in braking section of mountain city metro. Surface texture of brake pad and Vertical stiffness and lateral stiffness of fastener parameters have obvious effects on the generation of rail corrugation. And rail corrugation can be effectively suppressed, when regular hexagonal surface texture of brake pad is adopted, or when vertical stiffness of fastener is 36.2 MN/m, vertical damping of fastener is 2 800 N.s/m, lateral stiffness of fastener is 25 MN/m, lateral damping of fastener is 2 000 N.s/m.