| |
Internet outage?Sounds like there is an internet outage tonight. Boardman headend.
Looks like the self-service site is down too. Anybody else seeing it?
From what I’ve heard TV is still working. |
|
|
 dslwanter22 years on this site
Premium Member
join:2002-12-16
Mineral Ridge, OH
1 edit
1 recommendation |
Completely down. Dead in the fucking water. The entire boardman system. Line rings busy lmao. My kids are pissed. |
|
dslwanter
1 recommendation |
to xpxp2002
Back up @ 1030, my kids are forever grateful. |
|
| |
Wonder what happened. About an hour and a half seems too short to have been a fiber cut.
Either some kind of unexpected confluence of failures or a configuration change gone wrong. |
|
YuRi
Premium Member
join:2008-11-04
1 recommendation |
to xpxp2002
|
|
dslwanter22 years on this site
Premium Member
join:2002-12-16
Mineral Ridge, OH  ·Armstrong
Ubiquiti UniFi AP-LR
Ubiquiti EdgeRouter X SFP
|
to xpxp2002
No clue. Came in from wing night with the father in law (Iron Man- Girard get it if you haven't yet) at about 9. Soon as I walked in, got dad why is the internet down? Figured something happened with my network so stumbled down to the rack, everything seemed normal. ONT had connection to whever I terminate in Austintown. Rebooted everhing, got ONT signal but still sync no surf, so this screamed head end. Called the support line met with a busy signal. Hopped on Facebook and saw the local community group buizzing, AT&T people were kind of laughing lol, then saw this thread. All came back about a half hour ago. |
|
dslwanter
1 recommendation |
to xpxp2002
There are still ongoing issues this morning. In my drive into work I noticed one of the power injectors flashing red for the HFC network indicating it was offline. |
|
 OSIU
Premium Member
join:2003-11-12
Nowhere
2 recommendations |
to xpxp2002
There were a bunch of people in Medina having issues last night, both coaxial and fiber. Makes me glad I went with the local fiber provider......  |
|
dslwanter22 years on this site
Premium Member
join:2002-12-16
Mineral Ridge, OH ·Armstrong
Ubiquiti UniFi AP-LR
Ubiquiti EdgeRouter X SFP
|
said by OSIU:There were a bunch of people in Medina having issues last night, both coaxial and fiber. Makes me glad I went with the local fiber provider...... I'm venturing to say the issue is in Butter then if multiple systems are effected. This makes me question failover and other mid to high network structure. With the self-service site effected last night, I wonder if they were attacked. |
|
| |
The fact that the TV QAMs over fiber continued working suggests to me that it was a failure in Butler or a link between Butler and Boardman. The headend receives the local/regional TV feeds over separate links -- nowadays most likely private fiber circuits operated by a third party, like Verizon, AT&T, or Lumen (think MPLS-style "private networks" over telco infrastructure). Those feeds are likely not coming through Butler for other markets like Boardman and Medina. That also tells me that the headend connections down to the hubs and PFPs were not impacted.
I have not heard of any markets east of Boardman reporting impacts. That makes me think the physical link from Medina to Butler runs through Boardman/North Lima, and perhaps there was an equipment failure there, taking out Medina and Boardman. I'd be surprised if there isn't a direct path from Medina to Butler, though; only using Boardman as an alternate.
I really don't think it was a fiber cut due to the relatively quick recovery. If it were a fiber cut, they'd either re-route over an alternate path within minutes or it would have taken several more hours to see recovery waiting on a repair/splice in the field. Only scenario I can imagine where it would be a fiber cut is if in combination with some other failure. In other words, perhaps there is an alternate path for Boardman (and Medina) back to Butler, but due to a routing misconfiguration or undetected failure in the backup link, the primary failed and when the backup path was needed it didn't kick in, cascading into an actual loss of connectivity that was able to be quickly rectified with a config fix or rollback once the cause was identified. |
|
comp
Premium Member
join:2001-08-16
Evans City, PA ·Armstrong
|
to dslwanter
said by dslwanter:said by OSIU:There were a bunch of people in Medina having issues last night, both coaxial and fiber. Makes me glad I went with the local fiber provider...... I'm venturing to say the issue is in Butter then if multiple systems are effected. This makes me question failover and other mid to high network structure. With the self-service site effected last night, I wonder if they were attacked. Zero Issues over here in the Butler Area. I also an 99% sure each area has its own "CDN" There is no datacenter in Butler |
|
comp
1 edit |
to xpxp2002
said by xpxp2002:The fact that the TV QAMs over fiber continued working suggests to me that it was a failure in Butler or a link between Butler and Boardman. The headend receives the local/regional TV feeds over separate links -- nowadays most likely private fiber circuits operated by a third party, like Verizon, AT&T, or Lumen (think MPLS-style "private networks" over telco infrastructure). Those feeds are likely not coming through Butler for other markets like Boardman and Medina. That also tells me that the headend connections down to the hubs and PFPs were not impacted.
I have not heard of any markets east of Boardman reporting impacts. That makes me think the physical link from Medina to Butler runs through Boardman/North Lima, and perhaps there was an equipment failure there, taking out Medina and Boardman. I'd be surprised if there isn't a direct path from Medina to Butler, though; only using Boardman as an alternate.
I really don't think it was a fiber cut due to the relatively quick recovery. If it were a fiber cut, they'd either re-route over an alternate path within minutes or it would have taken several more hours to see recovery waiting on a repair/splice in the field. Only scenario I can imagine where it would be a fiber cut is if in combination with some other failure. In other words, perhaps there is an alternate path for Boardman (and Medina) back to Butler, but due to a routing misconfiguration or undetected failure in the backup link, the primary failed and when the backup path was needed it didn't kick in, cascading into an actual loss of connectivity that was able to be quickly rectified with a config fix or rollback once the cause was identified. There is not a Datacenter in Butler Butler is just the offices HQ They have DC's scattered around the areas which is why there Net Ops Engineers are required to travel 2-3 days a week |
|
dslwanter22 years on this site
Premium Member
join:2002-12-16
Mineral Ridge, OH ·Armstrong
Ubiquiti UniFi AP-LR
Ubiquiti EdgeRouter X SFP
|
said by comp:said by xpxp2002:The fact that the TV QAMs over fiber continued working suggests to me that it was a failure in Butler or a link between Butler and Boardman. The headend receives the local/regional TV feeds over separate links -- nowadays most likely private fiber circuits operated by a third party, like Verizon, AT&T, or Lumen (think MPLS-style "private networks" over telco infrastructure). Those feeds are likely not coming through Butler for other markets like Boardman and Medina. That also tells me that the headend connections down to the hubs and PFPs were not impacted.
I have not heard of any markets east of Boardman reporting impacts. That makes me think the physical link from Medina to Butler runs through Boardman/North Lima, and perhaps there was an equipment failure there, taking out Medina and Boardman. I'd be surprised if there isn't a direct path from Medina to Butler, though; only using Boardman as an alternate.
I really don't think it was a fiber cut due to the relatively quick recovery. If it were a fiber cut, they'd either re-route over an alternate path within minutes or it would have taken several more hours to see recovery waiting on a repair/splice in the field. Only scenario I can imagine where it would be a fiber cut is if in combination with some other failure. In other words, perhaps there is an alternate path for Boardman (and Medina) back to Butler, but due to a routing misconfiguration or undetected failure in the backup link, the primary failed and when the backup path was needed it didn't kick in, cascading into an actual loss of connectivity that was able to be quickly rectified with a config fix or rollback once the cause was identified. There is not a Datacenter in Butler Butler is just the offices HQ They have DC's scattered around the areas which is why there Net Ops Engineers are required to travel 2-3 days a week I gotcha each system is independent with separate data centers throughout? Level up to a backbone like level 3 or att? |
|
comp
Premium Member
join:2001-08-16
Evans City, PA ·Armstrong
|
comp
Premium Member
2023-Dec-1 8:31 pm
said by dslwanter:said by comp:said by xpxp2002:The fact that the TV QAMs over fiber continued working suggests to me that it was a failure in Butler or a link between Butler and Boardman. The headend receives the local/regional TV feeds over separate links -- nowadays most likely private fiber circuits operated by a third party, like Verizon, AT&T, or Lumen (think MPLS-style "private networks" over telco infrastructure). Those feeds are likely not coming through Butler for other markets like Boardman and Medina. That also tells me that the headend connections down to the hubs and PFPs were not impacted.
I have not heard of any markets east of Boardman reporting impacts. That makes me think the physical link from Medina to Butler runs through Boardman/North Lima, and perhaps there was an equipment failure there, taking out Medina and Boardman. I'd be surprised if there isn't a direct path from Medina to Butler, though; only using Boardman as an alternate.
I really don't think it was a fiber cut due to the relatively quick recovery. If it were a fiber cut, they'd either re-route over an alternate path within minutes or it would have taken several more hours to see recovery waiting on a repair/splice in the field. Only scenario I can imagine where it would be a fiber cut is if in combination with some other failure. In other words, perhaps there is an alternate path for Boardman (and Medina) back to Butler, but due to a routing misconfiguration or undetected failure in the backup link, the primary failed and when the backup path was needed it didn't kick in, cascading into an actual loss of connectivity that was able to be quickly rectified with a config fix or rollback once the cause was identified. There is not a Datacenter in Butler Butler is just the offices HQ They have DC's scattered around the areas which is why there Net Ops Engineers are required to travel 2-3 days a week I gotcha each system is independent with separate data centers throughout? Level up to a backbone like level 3 or att? Correct and I am 99% sure they do use Level 3 exclusively for backbone |
|
| |
Most of their interconnections are (unsurprisingly) in PIT-IX, and NYC. Peering is almost entirely with Zayo. » bgp.he.net/AS27364 |
|
| |
to OSIU
Weird...had no issues last night on Armstrong fiber north of the square in Medina. Guess I was lucky for a change. |
|
