This paper investigates joint design and optimization of both low density parity check (LDPC) codes and M-algorithm based detectors including iterative tree search (ITS) and soft-output M-algorithm (SOMA) in multiple-input multiple-output (MIMO) systems via the tool of extrinsic information transfer (EXIT) charts. First, we present EXIT analysis for ITS and SOMA. We indicate that the extrinsic information transfer curves of ITS obtained by Monte Carlo simulations based on output log-likelihood rations are not true EXIT curves, and the explanation for such a phenomenon is given, while for SOMA, the true EXIT curves can be computed, enabling the code design. Then, we propose a new design rule and method for LDPC code degree profile optimization in MIMO systems. The algorithm can make the EXIT curves of the inner decoder and outer decoder match each other properly, and can easily attain the desired code with the target rate. Also, it can transform the optimization problem into a linear one, which is computationally simple. The significance of the proposed optimization approach is validated by the simulation results that the optimized codes perform much better than standard non-optimized ones when used together with SOMA detector. This paper investigates joint design and optimization of both low density parity check (LDPC) codes and M-algorithm based detectors including iterative tree search (ITS) and soft-output M-algorithm (SOMA) systems via the tool of extrinsic information transfer (EXIT) charts. First, we present EXIT analysis for ITS and SOMA. We indicate that the extrinsic information transfer curves of ITS obtained by Monte Carlo simulations based on output log-likelihood steps are not true EXIT curves, and the explanation for such a phenomenon is given, while for SOMA, the true EXIT curves can be computed, enabling the code design. Then, we propose a new design rule and method for LDPC code degree profile optimization in MIMO systems. algorithm can make the EXIT curves of the inner decoder and outer decoder match each other properly, and can easily attain the desired code with the target rate. The significance of the proposed optimization approach is validated by the simulation results that the optimized codes perform much better than standard non-optimized ones when used together with SOMA detector.