The work aims to study the respective effect of thermal and phase transformation strain on carburizing quenching distortion, and analyze the influence of carburizing and quenching process on the distortion of gear alloy steel. The mechanism and deformation process of carburizing-quenching distortion on the C-shaped opening distortion sample made of gear alloy steel 17CrNiMo6 were numerically simulated with/without phase transformation consideration. The effect of the carburizing and quenching process on the distortion was quantified by orthogonal experiment. In order to verify the analysis results, the carbon content of the surface and distortion were determined by carburizing-quenching experiment. In the initial process, the thermal strain was about 2 times of the phase transformation strain. The order of F ratio of all process parameters in the orthogonal experiment from large to small was carburizing temperature (1.74), quenching temperature (1.546), carbon potential (1.448), preheating temperature (0.603) and oil temperature (0.473). The optimized process parameters were carburizing temperature 880 ℃, quenching temperature 790 ℃, preheating temperature 500 ℃, carbon potential 0.8%, and oil temperature 80 ℃. After optimization, the distortion rate was reduced by 28.5% and the distortion analysis results corresponded well with the experimental results. The thermal strain caused by thermal expansion takes the initiative to distort the sample and counteracts and exceeds the transformation strain. Carburizing temperature, carbon content and quenching temperature have a greater effect on distortion of gear alloy.