Sky shows up in point cloud

Hi,

This forum has been very helpful. Today I want to ask about why when I map the exterior of a building outdoors, some bits of sky show up in the point cloud generated by the ZED 2 (ROS) ? We know pixels in the sky should be calculated as having infinite distance and be excluded from the point cloud, but are there any tips for having sky points not show up? Thank you.