Although many methods have been applied to diagnose the gear fault currently, the sensitivity of them is not very good. In order to make the diagnosis methods have more excellent integrated ability in such aspects as precision, sensitivity, reliability and compact algorithm, and so on, and enlightened by the energy operator separation algorithm (EOSA), a new demodulation method which is optimizing energy operator separation algorithm (OEOSA) is presented. In the algorithm, the non-linear differential operator is utilized to its differential equation:Choosing the unit impulse response length of filter and fixing the weighting coefficient for inportant points. The method has been applied in diagnosing tooth broden and fatiguing’crack of gear faults successfully. It provides demodulation analysis of machine signal with a new approach.