The performance of adding additional inertial data to improve the accuracy and robustness of visual tracking is investigated. For this real-time structure and motion algorithm, fusion is based on Kalman filter framework while using an extended Kalman filter to fuse the inertial and vision data, and a bank of Kalman filters to estimate the sparse 3D structure of the real scene. A simple, known tar get is used for the initial pose estimation. Motion and structure estimation filters can work alternately to recover the sensor motion, scene structure and other parameters. Real image sequences are utilized to test the capability of this algorithm. Experimental results show that the proper use of an additional inertial information can not only effectively improve the accuracy of the pose and structure estimation, but also handle occlusion problem.