The differential equations of motion of a relativistic variable mass system are given. By using the invariance of the differential equations under the infinitesimal transformations of groups, the determining equations and the restriction equations of the Lie symmetries of a relativistic variable mass system are built, and the structure equation and the conserved quantity of the Lie symmetries are obtained. Then the inverse problem of the Lie symmetries is studied. The corresponding Lie symmetries are found according to a known conserved quantity. An example is given to illustrate the application of the result.