Displacement is a good descriptor of the structural behavior and safety status. However, measuring displacement of structures under dynamic excitations is still a challenging task. Videogrammetry shows great potential for dynamic displacement measurement, benefiting from its non-contact and cost-effective characteristics. Nevertheless, its all-weather performance has to be fully evaluated and improved when necessary before gaining wide applications. Therefore, long-term outdoor dynamic displacement monitoring tests using videogrammetry were carried out. A virtual displacement generation device was fabricated to generate dynamic displacements of various structures and the industrial digital camera was employed to monitor the virtual displacements. In this paper, virtual displacements of high-rise buildings measured under raining conditions were analyzed. It is found that raining imposes significant effects on the videogrammety, which noticeably decreases the accuracy of displacement measurement data. To widen the applicability of videogrammetry, adaptive filtering was proposed to de-noise the displacement measurement data. The results show that the adaptive filtering is well capable of decreasing/removing the measurement noises and enhancing the accuracy of displacement measurement data. It is helpful for the realization of all-weather videogrammetry for dynamic displacement monitoring of high-rise buildings.