With the high speed trend of the high precision machine tool spindle system, to study its stability in design process has become very important. Firstly, the instability mechanism of the hydrostatic spindle system was discussed and the stability characteristic of the high-speed built-in motorized hydrostatic spindle system was analyzed. Then, an optimization method for stability characteristic of the motorized spindle was suggested based on genetic algorithm. The objective was to maximize the stability evaluation index and the axial size parameters of the spindle system were selected as the design variables. The effective micro genetic algorithm was adopted. Eventually, higher stability was achieved. In the mean time, the modification of the parameters of complex structure was averted. As an example, the stability characteristic of a high-speed grinder motorized hydrostatic spindle system was analyzed and then optimized using this optimization method. The results indicated, the stability margin of the motorized spindle can be improved obviously by this method.