In order to solve the problem that the traditional frequency domain least mean square(FD-LMS)algorithm will lose efficacy with the increase of differential mode group delay(DMGD)when the algorithm is used for demultiplex-ing of the 6×6 mode division multiplexing(MDM)system,an improved FD-LMS demultiplexing algorithm is pro-posed.By improving the error signal calculation method,the convergence performance of the output signal of the equalization filter is improved,and the steady-state error of the algorithm is reduced.Besides,the equalization per-formance of the traditional FD-LMS algorithm is compared with the improved FD-LMS algorithm.Simulation results show that the improved FD-LMS algorithm has great advantage over the traditional FD-LMS algorithm in demulti-plexing performance on the premise that the computation complexity does not significantly increase.The optical signal to noise ratio(OSNR)penalty of the improved FD-LMS algorithm is 2.6 dB lower than that of traditional FD-LMS al-gorithm at a transmission distance of 80 km with DMGD is 50 ps/km.