Aiming at the power flow tracking performance optimization problem of modular multilevel converter (MMC)-unified power flow converter (UPFC) in novel power system, a Lyapunov nonlinear control strategy of MMC-UPFC under unbalanced grid is proposed in this paper. Firstly, according to the MMC-UPFC topology, the MMC-UPFC mathematical model is established and its internal characteristics are analyzed. Secondly, the positive and negative sequence Lyapunov controllers of MMC-UPFC under unbalanced conditions are designed and the stability is proved. Then, the signal is transmitted to the MMC by the carrier phase shift modulation after the sub-module voltage equalization control. Finally, the MMC-UPFC simulation system is built for simulation verification, the effects of Lyapunov control and PI control are compared to verify the correctness and effectiveness of the proposed Lyapunov control under two different operating conditions.