How to troubleshoot this network connection timeout?

Question

I have a strange network problem, something I've never seen before.

I can't reach a remote server from my local machine, but I can reach it from other machines outside my office network: https://raa.namecheap.com/

By reach, I mean HTTP, ping and traceroute.

The connection just times out. There are no response headers.

There is nothing special about my office network connection; it's a normal Comcast Residential account, no proxy, no firewalls, no url blocking, no nothing. It's a vanilla wired connection; essentially in the DMZ.

Context

Namecheap sends me a verification email with a link I need to visit that will not load from my office location:

https://raa.namecheap.com/ConfirmProfile.aspx?VerificationKey=xxx

Troubleshooting

I've tried the following:

• Namecheap tech confirms raa.namecheap.com is not blocking my IP
• Load a virgin Win10 VM from my office location, install fresh Chrome. Did not work
• Replaced my Comcast modem; got a new public IP assignment. Did not work
• Used several different DNS servers, including default Comcast. Did not work
• Connected successfully using kproxy.com
• Connected successfully from my remote server via SSH using Lynx
• Connected successfully from my phone's 5G connection

One of the many Namecheap techs I talked to said it may be a caching problem along one of the hops from my office to the destination server.

The Namecheap techs had to click the links for me to finish the verification, but the question remains, what in the world is going on?

EDIT

Here is a traceroute from a local Debian VM in my office. I don't know how to troubleshoot with SSH port forwarding, as suggested in the comments below.

MTR shows the same route I see from the Windows host:

EDIT

If someone would like to post an answer with troubleshooting ideas, I'll work through it and post my results. I really want to know what's going on. It's driving me nuts. If it can't be fixed, I can live with it, but I'd really like to know what is going on.

EDIT

Added a couple screenshots of modem firewall settings and internal modem ping/traceroute tests.

EDIT

Here is the tcptraceroute output, per the suggestion below.

Answer 1

Before you start changing everything on your own network you can check connectivity from Comcast routers by using their own route view server: ssh [email protected]

********************************************************************************
                       Comcast Backbone Route Server
        This route server is provided by Comcast National Engineering to provide 
visibility into the Internet routing table from the perspective of Comcast's 
network.  

Supported IPv4 Commands  
ping x.x.x.x <cr>       
traceroute x.x.x.x <cr> 
show bgp x.x.x.x <cr> 
show bgp x.x.x.x/y <cr> 
show bgp x.x.x.x/y longer-prefixes <cr> 

Supported IPv6 Commands       
ping ipv6 x:x:x::x <cr> 
traceroute ipv6 x:x:x::x <cr> 
show bgp ipv6 unicast x:x:x:x::x  <cr> 
show bgp ipv6 unicast x:x:x:x::x/y  <cr> 
show bgp ipv6 unicast x:x:x:x::x/y longer-prefixes <cr> 

Note: Due to high CPU utilization on this device, ping and traceroute results
may be unreliable.  This route server should not be used to measure network
performance as a result.

Login with username: rviewsxr

Location:   New York City
Network:  Comcast Route Server
********************************************************************************

Once connected you can see if the address is routable from the Comcast routers.

Some quick testing shows that the results you are seeing are not just specific to your network:

RP/0/RSP0/CPU0:route-server.newyork.ny.ibone#ping 198.54.117.244
Mon Aug  8 19:32:40.178 utc
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 198.54.117.244, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

Traceroute in case end node is blocking ICMP

RP/0/RSP0/CPU0:route-server.newyork.ny.ibone#traceroute 198.54.117.244
Mon Aug  8 19:38:00.349 utc

Type escape sequence to abort.
Tracing the route to 198.54.117.244

 1   *  *  * 
 2   *  *  * 
 3   *  *  * 
 4   *  *  * 
 <--SNIP-->
 30  *  *  * 
RP/0/RSP0/CPU0:route-server.newyork.ny.ibone#

The route exists in their tables, but there is something strange going on between Comcast and Incapsula (ASN 19551)

RP/0/RSP0/CPU0:route-server.newyork.ny.ibone#show bgp 198.54.117.0/24
Mon Aug  8 19:35:04.378 utc
BGP routing table entry for 198.54.117.0/24
Versions:
  Process           bRIB/RIB  SendTblVer
  Speaker          280867491   280867491
Last Modified: Jul 15 06:03:10.014 for 1y03w
Paths: (1 available, best #1)
  Not advertised to any peer
  Path #1: Received by speaker 0
  Not advertised to any peer
  19551 22612, (received & used)
    66.208.229.9 from 66.208.229.9 (68.86.1.48)
      Origin IGP, metric 0, localpref 275, valid, internal, best, group-best
      Received Path ID 0, Local Path ID 0, version 280867491
      Community: 7922:402 7922:3020
      Originator: 68.86.1.48, Cluster list: 96.109.22.250, 96.109.22.30
RP/0/RSP0/CPU0:route-server.newyork.ny.ibone#

From the looks of it, it is not a local network issue. You can try opening a ticket with Comcast, but you may have better luck letting Namecheap know that no XFinity/Comcast user can access their services and that they should let their providers (Incapsula) know.

Good luck!

Answer 2

It’s been years now, but I dealt with this exact same problem on a Comcast network. I noticed that certain IP addresses were unreachable. Just like you, traceroute would show several hops reachable and then just die in transit. The problem always appeared outside my network. But, it turns out it wasn’t.

It was the “Gateway Smart Packet Detection” setting in the modem. It needs to be disabled and it is on by default. However, I think this might be an older option and I think they also started disabling it by default because it was so problematic.

However, being the problem is exactly the same I would not be surprised if they reintroduced this feature in some shape or form. Especially if it is an SMC device.

If I remember right, power cycling the modem will solve the issue for a short period of time. If that’s your case, you can be sure it’s this feature, or it’s kin.

Here’s a screenshot of the setting I’m used to on the older device: https://routerhelp.net/tips-and-tricks/disable-smart-packet-inspection-on-comcast-smc-gateway/

Here’s references to a few problems it has caused: https://support.therapynotes.com/article/89-comcast-gateway-smart-packet-detection

And also:

https://ckdake.com/content/2008/disable-gateway-smart-packet-detection

If you’re running a static IP, you should also use the ‘Disable firewall for true static IP subnet only’ option as well. Or, barring all of this and depending on your modem, make sure it is in bridge mode when using a static IP - which requires a router behind it.

Answer 3

Traditional traceroute packets over UDP are filtered/blocked/dropped on the network.

You may want to try tcptraceroute to see a different picture.

I could see a few reasons why you couldn't reach the destination IP:

• firewall rule (locally, or somewhere else along the way/at destination network; you seem to have removed the "locally" option);
• misconfigured routing tables/BGP along the way;
• something else by Comcast.

The second one might be a routing loop, aka the ICMP packets get stuck in a loop, and traceroute timesout. Or maybe a router is lying, and doesn't actually know how to go further from there.

Sadly, I don't think you can force your traffic to go to a destination via a middle host (like you can on a map where you can go from A to B via C).

Maybe you could open a VPN connection to hosts further and further away that would bypass those routes altogether.

Have you tried having a new network interface using a different physical network? (aka via your phone hotspot, 5G network, etc.)