Hi @NNSkelly,
One option is to set the camera as static in Unity in the ZEDManager, which should prevent drifting issues.
That being said, to check if the IMU is at fault, you can use ZED Sensors Manager (in the tools folder of the SDK), put your camera on a stable surface, and check the position and rotation. If you see unexpected movement, you can reset the calibration of the camera, as described in this forum post.