This article proposes a method of constructing a ping-pong system via markerless real rackets in collaborative augmented reality. Except a pair of video cameras, without any other sensors or artificial markers, users can use real rackets to hit virtual ping-pong ball on a virtual table and interact with remote partners in augmented reality scene just as they were playing ping-pong in the same place. First, the real racket can be detected and tracked in real-time in the video captured by a single camera in each site. By 3D registration, the real racket can seamlessly interact with the virtual ping-pong ball and table. Then, a communication scheme is designed for the consistent perception between users in collaborative augmented reality ping-pong system. To achieve real-time interaction, the whole method is implemented in a parallel computing environment through multi-core processors. Experimental results demonstrate that our system can provide consistent perception and natural user interaction with low latency and high precision.