The only time I've ever had a pannier pop off was on the Middlewood way, I think there's a concrete farm road going over which stands a little proud, it had my spare battery in too, luckily no damage to the battery.
It gets pretty wet and muddy down there around the A6 doesn't it ?