Concerning the harmonic instability problem of LCC-HVDC transmission system, especially the phenomenon of harmonic instability caused by complementary resonance, the harmonic impedance models of AC system impedance, AC filter and converter transformer are established based on the modulation theory of converter switching function. The influence of the number of generators and filter groups on the resonance points of the system is analyzed. A harmonic instability suppression strategy is further proposed to destroy the resonance point by changing the operation mode of the system，and the influence of DC control mode on the harmonic is analyzed. The standard CIGRE model is established in PSCAD/EMTDC to simulate and verify the proposed suppression strategy. The simulation results verify the correctness and effectiveness of the proposed strategy.