Data obtained from the mobile SOUSY VHF radar at And(ya/Norway in summer 1987 have been used to study the nonlinear interactions between planetary waves, tides and gravity waves in the polar mesosphere, and the instability of background atmosphere above the mesopause. It is observed that 35-h planetary wave, diurnal, semidiurnal and terdiurnal tides are the prominent perturbations in the Lomb-Scargle spectra of the zonal wind component. By inspecting the frequency combinations, several triads are identified. By bispectral analysis it is shown that most bispectral peaks stand for quadratic coupling between tidal harmonics or between tide and planetary or gravity wave, and the height dependence of bispectral peaks reflects the variation of wave-wave interactions. Above the mesopause, the occurrence heights of the maximum L-S power spectral peaks corresponding to the prominent wave components tend to increase with their frequencies. This may result from the process in which two low frequency waves interact to generate a high frequency wave. Intensities of the planetary wave and tides increase gradually, arrive at their maxima, and then decay quickly in turn with increasing height. This kind of scene correlates with a "chain" of wave-wave resonant interactions that shifts with height from lower frequency segment to higher frequency segment. By instability analysis, it is observed that above the mesopause, the Richardson number becomes smaller and smaller with height, implying that the turbulent motion grows stronger and stronger and accordingly the background atmosphere more and more instable. It is suggested that the wave-wave sum resonant interaction and the wave dissipation due to instability are two dominant dynamical processes that occur in the mesopause region. The former invokes the energy transfer from lower frequency waves to higher frequency waves. The latter results in the heating of the atmosphere and accelerating of the background flow.