The dynamical properties of fractional-order Duffing–van der Pol oscillator are studied, and the amplitude–frequency response equation of primary resonance is obtained by the harmonic balance method. The stability condition for steady-state solution is obtained based on Lyapunov theory. The comparison of the approximate analytical results with the numerical results is fulfilled, and the approximations obtained are in good agreement with the numerical solutions. The bifurcations of primary resonance for system parameters are analyzed. The results show that the harmonic balance method is effective and convenient for solving this problem, and it provides a reference for the dynamical analysis of similar nonlinear systems.