The Free-floating Flexible Dual-arm Space Robot is a highly nonlinear and coupled dynamics system. In this paper, the dynamic model is derived of a Free-floating Flexible Dual-arm Space Robot holding a rigid payload. Furthermore, according to the singular perturbation method, the system is separated into a slow subsystem representing rigid body motion of the robot and a fast subsystem representing the flexible link dynamics. For the slow subsystem, based on the second method of Lyapunov, using simple quantitative bounds on the model uncertainties, a robust tracking controller design is used during the trajectory tracking phase. The optimal control method is designed in the fast subsystem to guarantee the exponential stability. With the combination of the two above, the system can track the expected trajectory accurately, even though with uncertainty in model parameters, and its flexible vibration gets suppressed, too. Finally, some simulation tests have been conducted to verify the effectiveness of the proposed methods.