We study the quantum phase transition from a superfluid to a Mott insulator of ultracold atoms in a three-dimensional optical lattice with adjustable filling factors.Based on the density-adjustable Bose-Einstein condensate we prepared,the excitation spectrum in the superfluid and the Mott insulator regime is measured with different ensemble-averaged filling factors.We show that for the superfluid phase,the center of the excitation spectrum is positively correlated with the ensemble-averaged filling factor,indicating a higher sound speed of the system.For the Mott insulator phase,the discrete feature of the excitation spectrum becomes less pronounced as the ensemble-averaged filling factor increases,implying that it is harder for the system to enter the Mott insulator regime with higher filling factors.The ability to manipulate the filling factor affords further potential in performing quantum simulation with cold atoms trapped in optical lattices.