When the large no-load transformer is switched to connect with the power grid, magnetizing inrush current may be resulted for the residual flux exists in the iron core, and thus the transformer perhaps fail to connect with the power grid. A new method for analyzing and measuring residual flux in the closed iron core is proposed in this paper, based on the positive and negative DC voltage excitation method. The principle of residual flux generation is introduced, and then considering the hysteresis characteristic of magnetic material a toroidal iron core simulation model is established based on Jiles-Atherton (J-A) model. By applying two DC voltage with same magnitude but different polarity to the winding on the closed toroidal iron core, two different response currents can be obtained. Based on coupled magnetic field and circuit transient analysis method, the residual flux direction can be determined, and then the relationship between the response current and the residual flux can be also found. This method can accurately measure the value and direction of the residual flux when it has little effect on the initial residual flux in iron core. Lastly an experiment platform is established, and the simulation results are verified.