Due to the influence by human factors on the stochastic subspace identification results, hierarchical clustering algorithm is introduced to overcome this shortcoming. spurious modes resulting from noise or model redundancy can be removed according their mode energy or mode similar index which describes the similarity between modes obtained by two different algorithm; the hierarchical clustering method is introduced to pick up the system modes, the results can be classified into several categories according the similarity between the results which can be calculated by eigenfrequencies, damping ratios, mode shapes and mode energy, some categories will be selected if the number of its elements is large enough. A numerical example and an experimental example are presented to demonstrate the efficacy of the method.