In order to improve the surface quality of complex surface in slow tool servo turning, the tool compensation algorithm was optimized. In view of the problems that normal compensation algorithm can easily lead to the decrease of the dynamic performance of X-axis and large interpolation error in Z-direction compensation algorithm, a geometric compensation algorithm based on coordinate transformation was proposed in this paper. Coordinate transformation can improve the accuracy of the solution and simplify the algorithm. By using the geometric transformation relationship, the compensation component of X-axis could be concentrated on the Z-axis, which not only ensured the dynamic performance of X-axis, but also reduced the interpolation error. Taking the toric surface as an example, the tool compensation algorithm proposed in this paper was simulated and verified by experiments. The simulation results showed that the velocity of X-axis fluctuates greatly under the normal compensation algorithm, while the X-axis can keep uniform motion under the algorithm proposed in this paper. In the tool compensation link, compared with the algorithm proposed in this paper, the interpolation error under Z-direction compensation algorithm was larger, and the maximum interpolation error was more than 0.015 mm. The experimental results showed that the value of surface roughness of the toric surface was the largest under the normal compensation algorithm (Ra=0.112 μm), which was much larger than that under the Z-direction compensation algorithm and the algorithm proposed in this paper. However, under the Z-direction compensation algorithm and the algorithm proposed in this paper, the value of surface roughness of the toric surface was similar (Ra=0.066 μm and Ra=0.062 μm respectively), which indicates that the tool compensation algorithm has little effect on the surface roughness on the premise of ensuring the dynamic performance of X-axis. The values of PV obtained under the normal compensation algorithm, the Z-direction compensation algorithm and the algorithm proposed in this paper was 16.9 μm, 13.8 μm and 8.8 μm respectively. Compared with normal compensation algorithm and Z-direction compensation algorithm, the accuracy of toric surface was improved by 92.0% and 56.8% respectively under the algorithm proposed in this paper, which shows that the tool compensation algorithm proposed in this paper can improve the surface machining quality.