To improve the precision, the antinoise and the real-time performance of VCO nonlinearity detection methods presently, a new method based on sliding Fractional Fourier Transform (FrFT) for instantaneous frequency estimation was proposed. This method overlapped sliding intercepts the estimated signal to improve the antinoise first. Then get middle-time’ instantaneous frequencies of all interceptive signals by sliding FrFT, and improve the real-time performance by golden section and chaos optimization method to search the optimal order of sliding FrFT. Last get a more precise instantaneous frequency by cubic splines interpolation between those middle-time’ instantaneous frequencies. Experiment results show the proposed method has better performance than existing methods at Signal Noise Ratio (SNR) from -5dB to 5dB with a suitable calculation.