Recently, as recognizing emotion has been one of the hallmarks of affective computing, more attention has been paid to physiological signals for emotion recognition. This paper presented an approach to emotion recognition using ElectroCardioGraphy (ECG) signals from multiple subjects. To collect reliable affective ECG data, we applied an arousal method by movie clips to make subjects experience specific emotions without external interference. Through precise location of P-QRS-T wave by continuous wavelet transform, an amount of ECG features was extracted sufficiently. Since feature selection is a combination optimization problem, Improved Binary Particle Swarm Optimization (IBPSO) based on neighborhood search was applied to search out effective features to improve classification results of emotion states with the help of fisher or K-Nearest Neighbor (KNN) classifier. In the experiment, it is shown that the approach is successful and the effective features got from ECG signals can express emotion states excellently.