GNMI stats proved very helpful to keep an eye on the bandwidth shifting around

Work completed, traffic is currently bridged through the two spine switches over the AEs from the row C/D virtual-chassis and the CR interfaces connected to the Spines are working as VRRP gateway.

There is possibly a variant of option 1:

In T370164#9993574, @ayounsi wrote:

Or we could just use a IPv6 /64 and stop worrying about space :)

In T370164#9989347, @ayounsi wrote:

We will need to migrate the whole range to a new prefix :( Running 2 ranges is going to be a pain long term, and would need much more work on the automation side than a migration.

In T369825#9992536, @VRiley-WMF wrote:

There is another rack in row B that we can use for this server. However, moving it to another rack will require an IP change when I last confirmed it.

Though there didn't seem to be a problem afterwards, the timing makes me think of T365997: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 -lsw1-f2-eqiad

In T368513#9938867, @fgiunchedi wrote:

Those are SSH probes from local prometheus hosts indeed, in this case the probe consists of a TCP connection reading the SSH banner, and then closing the connection, HTH!

Upgrade completed, all hosts back online and pinging ok. Thanks all for the assistance!

Switch upgrade complete, all looks good hosts are online and responding to ping again. Thanks for the assistance!

Switch upgraded successfully and all hosts back online/pinging. Thanks everyone for the assistance!

In T365995#9968809, @Marostegui wrote:

@cmooney got to be closed?

Closing task - is a duplicate work was completed under T365169

In T360356#9950740, @wiki_willy wrote:

ifconfig

I think the work on this can be done in tandem with the review of the setup in T367203: Sub-optimal cloud routing for WMCS in eqiad when link fails.

In T369011#9968167, @JMeybohm wrote:

I've disabled BGP for this node for now.

Switch upgrade completed without issue. All connected hosts are back online and responding to ping now, thanks all for the help.

In T369011#9948452, @JMeybohm wrote:

I've deleted the node from the k8s API as a required istio update would not finish successfully because it was waiting for the deamonset to be scheduled on the broken node. The node should auto-join the cluster (cordoned) when it comes back online.

Seems like a great tool, but we are going to move forward with pulling these stats using gnmic after successfully testing it under T326322. If we find any blockers that gNMI can't cover we can revisit using junos_exporter but hopefully that won't be needed. Future gnmic pipeline development will be tracked in T369384: Productionize gnmic network telemetry pipeline

I'm going to close this task now, the current gnmic collection is providing what we need in terms of the queue-stats for observing how QoS operates, and it seems best to track future, general improvements to our network telemetry in a separate task (see bleow).

Bit of an update on this one. We had a problem recently after lvs2011 was rebooted which is related, which we need to address.

In T362392#9955949, @ayounsi wrote:

Adding bfd yes; returns bird: /etc/bird/bird.conf:90:1 Multihop BGP with BFD requires specified local address
Adding local fe80::2022:22ff:fe22:2201 requires an interface stanza, which is not possible with dynamic tapX interfaces.

All seems good with the policy changes now, closing task.

All is working well on the test-host. Well puppet was giving me a headache but I just skipped all that :)

@Papaul sorry I meant to get back to you sooner. I've made decent progress on T369106 and managed to test reimage working ok in one of the new racks with the test server, so we are in a good position.

In T360789#9941103, @Papaul wrote:

All the cabling is done. I am leaving this task open so when we move the console cables from asw-c*/d*-codfw to ssw1-* and lsw1-* I can update the task and resolve it.

Ok change merged, we are now announcing codfw ranges from eqord again:

cmooney@cr2-eqord> show route advertising-protocol bgp 192.80.17.197

So one thing I noticed is that we are not getting the stats for LAG/ae interfaces with the current setup, nor routed sub-interface stats.

Switch is back up, all looks good at first glance from the network side.

Gonna close this one as the design is finalised, see detail on wikitech here:

Also @Jhancock.wm when next on site can you check the mgmt / idrac connection for this one? It doesn't seem to be trying to get an IP by DHCP, and the old IP from when it was mw2289 isn't working either.

@Jhancock.wm can you confirm what position in the rack the server is in?

All seems ok following the increase:

In T367512#9945244, @Jhancock.wm wrote:

@cmooney got sretest2002 on lsw-d4, ports 44 and 45. 10G card.

So the change to the timeout has made a big difference, but there are still some small gaps:

In T363341#9936269, @Jclark-ctr wrote:

cloudcephosd1039
2nd cable serial#20220008 port 1
cloudcephosd1040
2nd cable serial#20220043 port 5
cloudcephosd1041
2nd cable serial#20220011 port 7

Folks just FYI I've pushed the time here back an hour if that's ok, seems to suit most best.

I may have spoken too soon when I said things were working fine. It seems in codfw since the change we are only getting stats some of the time:

@fgiunchedi I was perhaps a little cheeky and merged this, but it was clear the volume of new metrics was well within what you'd said before was ok. Everything working nicely I'm glad to say.

In T368544#9933423, @ayounsi wrote:

An ip route 0/0 rule would be needed to "clamp" the outbound MTU or MSS (using mtu lock, or advmss for example) on any client or realserver with a jumbo MTU.

Thanks all for the help with this one!