Based on the analysis and comparison of protection configuration difference between UHVDC and the HVDC engineering, it is pointed out that the IIR digital filter is used for data preprocessing in converter transformer protection for UHVDC engineering. In this paper, from two perspectives of filter type and filter order, the filter parameter is designed by means of filter aided design tool. The effect on transient performance of converter transformer and connecting line incremental differential protection is analyzed, which is caused by using IIR digital filter for data preprocessing. On the premise of satisfying the low frequency characteristic of relay protection system, simulation results show that the 4 order Butterworth low-pass IIR filter is used as the optimum, which can meet the transient performance requirements of relevant specifications.