In this paper, the dispersion curves of the Rayleigh wave and Love wave were extracted from the seismic noise records of 25 broadband stations of the Fujian Seismic Network, and inverted for the lithosphere velocity structure. Furthermore, the velocity model was verified by the seismic explosion observations. Our results indicate that the resolution of the lithosphere velocity structure obtained by this method is good in the shallow part, but in the deep part, inversion accuracy for the wave velocity structure is low, which is caused mainly by the small inter-station distance chosen in the paper. Thus the wave dispersion curves have high accuracy in the short-period part, but the warp of the wave dispersion curve in long-period part is large. Considering the results from both the noise inversion and the traditional inversion, we finally present a new velocity model, and the theoretical travel time calculated with the new model matches the explosion travel time very well.