When the error of observation model is uncertain in the established lithium battery nonlinear system, the accuracy and stability of filter estimation will be affected, and the estimation results will diverge in serious cases. To solve this problem, a robust UKF algorithm is proposed based on the variational Bayesian adaptive filtering method. Firstly, the algorithm constructs a virtual observation noise to compensate the error of the observation model, and uses the inverse Wishart distribution to model the covariance of the virtual observation noise. Secondly, in the process of variational iteration, the joint posterior probability estimation of the covariance of system state and virtual observation noise is realized, which makes the estimation result approximate to the real distribution adaptively. Finally, the unscented Kalman filter is used to update the system state. The simulation results based on the non-linear model of lithium potassium manganate battery show that the algorithm has good accuracy, tracking speed and robustness in estimating the state of charge of lithium battery.