This paper aims to improve the machining quality of hardened 12Cr2Ni4A steel, improve the machining accuracy and eliminate the residual stress on the workpiece surface. The hardened 12Cr2Ni4A alloy steel was processed by ordinary grinding (OG), ultrasonic vibration assisted grinding (UVAG) and ultrasonic vibration assisted ELID grinding (UVAEG) respectively. The surface roughness and residual stress of the workpiece machined under the three grinding modes were analyzed. The result was that the surface roughness of workpiece in UVAG and UVAEG was lower than that of OG, while the surface quality of workpiece after UVAEG was the highest. Compared with OG, the surface roughness decreased by 66% and 41% compared with UVAG; the residual stress measurement of workpiece surface showed that the surface of workpiece was residual tensile stress after OG, while residual compressive stress was produced after UVAG and UVAEG. The residual compressive stress of workpiece surface after UVAEG was about 30% higher than that of UVAG; in OG machining, the residual tensile stress had been increasing with the increase of the grinding depth, while the residual compressive stress of UVAG and UVAEG generally showed a decreasing trend. After the grinding depth reached 22.5 μm, the residual compressive stress on the machined surface of UVAG was transformed into residual tensile stress. The surface residual stress of UVAG and UVAEG increased with the increase of ultrasonic amplitude, and the surface residual stress of UVAEG increased with the increase of duty cycle. The conclusion is that UVAEG can obtain smaller surface roughness and larger surface residual compressive stress.