It has been found that the brittle material, monocrystalline silicon, can be machined in ductile mode in nanoscale cutting when the tool cutting edge radius is reduced to nanoscale and the undeformed chip thickness is smaller than the tool edge radius. In order to better understand the mechanism of ductile mode cutting of silicon, the molecular dynamics (MD) method is employed to simulate the nanoscale cutting of monocrystalline silicon. The simulated variation of the cutting forces with the tool cutting edge radius is compared with the cutting force results from experimental cutting tests and they show a good agreement. The results also indicate that there is silicon phase transformation from monocrystalline to amorphous in the chip formation zone that can be used to explain the cause of ductile mode cutting. Moreover, from the simulated stress results, the two necessary conditions of ductile mode cutting, the tool cutting edge radius are reduced to nanoscale and the undeformed chip thickness should be smaller than the tool cutting edge radius, have been explained.