This letter investigates a delay op-timization problem in device-to-device (D2D) networks where users have pre-downloaded subfiles following a coded caching rule. Due to acquiring rest subfiles, users may suffer both receiving and transmitting delays. To achieve the delay minimization, we first pro-pose a delay-aware mode-selection strategy to adaptively choose multicast or D2D commu-nications mode to reduce the receiving-caused delay. By matching these transmit modes with distinct subfile sizes, we further formulate a min-max optimization problem to minimize the delivery delay. Finally, numerical results prove that the proposed scheme outperforms existing ones in terms of both receiving and transmitting delays.