Once, during Sauron's war with the Elves in the Second Age
Gil-galad and the Númenóreans were holding the Lhûn in desperate defence the Grey Havens, when in the very nick of time the great armament of Tar-Minastir came in; and Sauron's host was heavily defeated and driven back.
Unfinished Tales Book 2 The Second Age Chapter 4: "The History of Galadriel and Celeborn"
This was about the year 1700 of the Second Age, and Sauron was pushed out of Eriador not long after this, when the Númenóreans entered the fight.
After the establishment of Gondor and the North Kingdoms, Sauron's forces were generally held at bay by them; Galdor notes during the Council of Elrond that Gondor has, so far, been able to keep the Havens safe:
Only the waning might of Gondor stands now between him and a march in power along the coasts into the North; and if he comes, assailing the White Towers and the Havens, hereafter the Elves may have no escape from the lengthening shadows of Middle-earth.
Fellowship of the Ring Book II Chapter 2: "The Council of Elrond"