The dynamic response of the forced vibration of a beam with one end clamped and the other end with bolted joint boundary condition is studied with the nonlinear force-displacement characteristic of the joint modeled using Iwan model for studying the effect of nonlinearities in the bolted joint on the dynamic response of the beam, based on mode transfer principle. The mode transfer between different states caused by the hysteretic boundary condition is treated adopting Relative Mode Transfer Method. The effects of the coupling between modes, number of the Jenkin's elements in the Iwan model, amplitude of the external driving force, modal damping and stiffness of the joint on the response are discussed through the frequency response of the beam's end. It is found the new approach has good convergence when dealing with vibration problem of continuous system with hysteresis boundary condition, the effect of the nonlinearity on the dynamic response is finite, one-period motion and weak frequency-shift are the only responses in the specified sweep range, and the frequency-shift is sensitive to the stiffness of the bolted joint.