This paper aimed at investigating the sideband distribution law caused by the sun/ring gear tooth crack fault in the planetary gear system. A loaded tooth contact analysis method was proposed, which can simulate the tooth crack fault under the actual crack morphology, and can simulate the output shaft installation error by changing the center distance. Based on this method, the time-varying mesh stiffness of all the gear pairs in the planetary gear system were solved and expressed at the same time scale considering the mesh phases. Meanwhile, this paper also proposed a hybrid dynamic model of the planetary gear system, the gear components were simulated based on the lumped parameter method, and the shafts were simulated based on the Timoshenko beam theory. By introducing the time-varying mesh stiffness into the dynamic model as vibration source, the dynamic response was derived. Through comparison with the experiment, it was found that by considering the sun/ring gear tooth crack fault and the output shaft installation error, the sideband distribution law of the simulated result was highly consistent with the experiment results. Finally, the sideband distribution laws of the sun/ring gear tooth crack fault under the output shaft installation error condition were summarized. This article is useful for monitoring and identifying the gear tooth crack fault of the planetary gear system.