The large-scale development and utilization of renewable energy has made the proportion of inverters in the power grid higher and higher, and the harmonic interaction between the inverter and the power grid has become more frequent and complex. Aiming at the problem that the traditional modeling method cannot reflect the inter-harmonic interaction coupling between the inverter and the power grid, the three-phase grid-connected inverter is modeled based on the harmonic state space (HSS) theory. This model takes into account the multiple frequency harmonics on the AC and DC sides, and has high accuracy. Based on the HSS model, the input-to-output harmonic transfer function matrix is derived, the AC/DC harmonic coupling admittance amplitude diagram of the inverter is established, and the AC/DC harmonic coupling relationship is explored. Then, a small-signal HSS modeling method for three-phase grid-connected inverters suitable for closed-loop control systems is proposed, which evaluates the stability of the system and predicts the oscillation frequency by calculating eigenvalues. Finally, the accuracy of the established HSS model and the correctness of the system stability analysis are verified by comparing with the simulation results of MATLAB/Simulink.