Due to the complex line conditions and train service conditions, the rail corrugation problem of a mountainous city metro is obvious. The metro operation and maintenance face great challenges. Based on the viewpoint of friction self-excited vibration, the work aims to study the friction self-excited vibration characteristics of wheelset-rail system under traction and braking conditions in the long steep grade section of a mountainous city metro. It is suggested that the friction self-excited vibration caused by saturated creep force between wheelset-rail system may induce the generation of rail corrugation. Firstly, in combination with the field investigation of the long steep grade section of a mountainous city metro, the vehicle-track system dynamics models under the uphill traction and downhill braking conditions were constructed. The vehicle-track system dynamics characteristics under the traction and braking conditions were explored. Then, the finite element models of wheelset-track-traction and wheelset-track-brake systems were constructed. The friction self-excited vibration characteristics of the wheelset-rail system under traction and braking conditions were studied by complex eigenvalue method. The causes of rail corrugation under traction and braking conditions in the long steep grade section of a mountainous city metro were revealed. Finally, by the least square method and particle swarm optimization algorithm, the optimal solution of vehicle-track parameters was obtained to suppress rail corrugation. It was found that the wheelset-track-traction system had an unstable vibration frequency of 369.93 Hz. Its modal vibration occurred in the outer rail, and the real part of the complex eigenvalue was 19.36. The wheelset-track-brake system had an unstable vibration frequency of 443.92 Hz. Its modal vibration occurred in the outer rail, and the real part of the complex eigenvalue was 4.970 5. Compared with wheelset-track-brake system, the real part of the complex eigenvalue of wheelset-track-traction system is larger, so the train is more likely to generate friction self-excited vibration in traction condition than in braking condition. The result is consistent with the field investigation that rail corrugation mainly occurs in the outer rail, and it is more likely to occur and more serious in the traction section than in the braking section. The results show that, in the same long steep grade section of a mountainous city metro, rail corrugation often occurs in the section that is the intersection of the circular curve section and easing curve section with slope changing greatly. Rail corrugation is more likely to occur and more serious in the traction condition than in the braking condition. The self-excited vibration of traction friction and wheelset-rail friction may be the main cause of rail corrugation in the traction section, and the self-excited vibration of braking friction and wheelset-rail friction may be the main cause of rail corrugation in the braking section. The former is more likely to happen than the latter. When the braking pressure is set to 5 kN under the braking condition, the generation of rail corrugation in the braking section can be effectively inhibited. When the traction running speed is 80 km/h under the traction condition, the vertical stiffness is set as 55 MN/m, the lateral stiffness is as 25 MN/m, the generation of rail corrugation in the traction section can be effectively inhibited.