The presence of array imperfection and mutual coupling in sensor arrays poses several challenges for development of effective algorithms for the direction-of-arrival (DOA) estimation problem in array processing. A correlation domain wideband DOA estimation algorithm without array calibration is proposed, to deal with these array model errors, using the arbitrary antenna array of omnidirectional elements. By using the matrix operators that have the memory and oblivion characteristics, this algorithm can separate the incident signals effectively. Compared with other typical wideband DOA estimation algorithms based on the subspace theory, this algorithm can get robust DOA estimation with regard to position error, gain-phase error, and mutual coupling, by utilizing a relaxation technique based on signal separation. The signal separation category and the robustness of this algorithm to the array model errors are analyzed and proved. The validity and robustness of this algorithm, in the presence of array model errors, are confirmed by theoretical analysis and simulation results.