'Fairly rare' weather event leaves Victorian bay unsuitable