Based on the comparison of results between the shaking table test and finite element analysis, the method of nonlinear finite element analysis of brick masonry structure is discussed, and the dynamic performance of a typical rural dwelling model without structural column were studied. The results showed that (1) The plastic damage model can be used to approximately simulate the mechanical behaviors of masonry structure , and the results of the finite element simulation were consistent with that of the test. (2) The typical rural house (strength class of brick and mortar are respectively Mu15 and M10) was able to withstand earthquake action corresponding to peak acceleration 0.3g, but would be seriously damaged and was inferior to the seismic requirements for large earthquake. (3) Importantly, reducing strength class of mortar or brick had great influence on masonry strength and the seismic performance of structure. In conclusion, in order to reduce earthquake damage, both the design and the construction process of self-built dwellings should be included in national standardized management.