The work aims to study the formation mechanism of the polygonal wear of high-speed train wheels and the effect of wheel-rail system structural parameters on the wheel polygonalization. Based on the theory that wheel-rail system frictional self-excited vibration caused by the saturated wheel-rail creep force could lead to wheel polygonal wear, a solid model including wheel, rail, sleepers and track beds was established. The solid model was imported into the finite element software ABAQUS. The rail and sleepers were simulated by point-to-point massless spring damping unit and the sleeper and the track bed were connected by binding constraint. Then the point-to-ground massless spring damping unit was used as the bottom support of the track bed. The stability of the wheel-rail system when the brake slip occurred on the high-speed railway lines was studied by the complex eigenvalue method. Under the effect of saturated creep force, the unstable vibration frequency of the wheel-rail system was f=495.01 Hz, the wheel was prone to generate 18-degree polygonalization. Within a certain range, the vertical stiffness of the fastener did not play a significant role in suppressing the wheel polygonalization. In order to suppress the wheel polygonalization, properly increasing the vertical damping of the fastener can suppress the frictional self-excited vibration of the wheel-rail systemeffectively. Different eccentric forms hardly had effects on the unstable vibration of the wheel-rail system. On a high-speed line, the braking slippage of the train can easily cause the wheel polygonalization. The wheel polygonalization can be effectively restrained by raising the vertical damping of the fastener.