The coupling between torsion vibration and the nonlinear friction can induce self-excited vibration in propeller shaft systems. In order to obtain the excitation mechanism and basic characteristics of the friction induced vibration, a nonlinear dynamic model of the friction-induced self-excited torsion vibration is formulated. The dynamic friction characteristic is represented by Stribeck curve. The equation of motion is derived using Lagrangian approach in conjunction with the modal superposition method. Using the multiple scales method, the critical parameter expression and the basic characteristics of self-excited vibration is then given out. The numerical method is also used to analyze the dynamic responses and effects of the physical parameter changes on the self-excited oscillation. The research method and results may provide theoretical references for the structure design of propeller shaft systems.