While doing some more research I came across this article on DSL wiring issues. The key part of the article for my case was the Signal to Noise Ratio (SNR):
Signal-to-noise ratio (SNR) is another figure you can find in your
ADSL stats of your home / corporate router (show dsl interface
atm(something) for Cisco). This number also expressed in dBs and
describes the relation between your speaking strength (signal) and
street’s noise strength (noise). The higher this number the better
since your voice outperforms the noise.
- 10dB and below is bad
- 11db – 20dB is OK
- 20dB – 28dB is excellent
- 29dB and above is outstanding
I started to monitor my SNR and saw that when the connection started dropping the SNR was dropping below 8dB right before. I tested hooking the router up directly to the phone line box and found my SNR to be 18.1dB without the line in the house.
While testing the outside connection I did see it drop by 2dB at one point and figured out that it was due to me putting my laptop next to the phone cable. That made me realize how sensitive the line was to noise.
Armed with this information I headed up to my wiring closet and did a few things:
- Wrapped the wire that was not wrapped in the CAT5 sheath with electrical tape
- Mounted the router on the wall of the closet instead of having it sitting on a server
- Made sure all the extra cable going to the router was hung up and away from any power cords
Having done all this the SNR is 18dB (down 0.1dB from outside) which is great. So thanks to MaQleod for stating the wire gauge shouldn't be the issue since that started me on the right track. Thanks for the other answers which also seem to point to noise being the issue, but it wasn't water, fumes, or the termination (good ideas though).