The interaction between new energy grid-connection and power electronic equipment in novel power system is easy to cause wideband oscillation, which poses a serious threat to the security of power grid. In this paper, the reliable detection method of oscillating signal in wideband measurement is studied. Aiming at the serious false alarm and missing detection problems in the peak search method of windowed discrete Fourier transform (DFT), it is proposed to use a filter constructed by the spectrum of the window function to filter the windowed DFT sequence and then conduct spectral peak detection, that is, an improved detection algorithm based on frequency domain matched filtering. The theoretical analysis shows that the matching filter not only enhances the signal-to-noise ratio of the detection decision, but also significantly improves the spectral characteristics of the window function. Tests are conducted on the windowed DFT of typical Hamming windows and Hann windows. Simulation results in static, dynamic and noisy environments show that the matched filter not only successfully eliminates false alarms and realizes reliable detection, but also significantly improves the accuracy of signal detection frequency estimation, which has important engineering application value.