SlideShare a Scribd company logo

100Gbps Core Network Deployment in an African Network - Mark Tinka

•
•
MyNOG
MyNOG

This document summarizes the network upgrade of two core points of presence (PoPs) in Johannesburg and Cape Town, South Africa to 100Gbps connectivity. The upgrades involved replacing the Cisco 6800-X Ethernet switches with Arista 7508E switches and upgrading the Cisco CRS routers with 100Gbps line cards. This provided significantly more bandwidth capacity and addressed growth concerns in these major markets. The changes required overcoming challenges around optics incompatibility, lack of experience with 100Gbps deployments in Africa, logistics and customs clearance, vendor training on Arista, and data center modifications. Ultimately, the results showed comparable CLI experience between Cisco IOS and Arista EOS, and introducing a third vendor was not seen

Related slideshows

Open Connect Appliances - Jocelyn Ooi
Open Connect Appliances - Jocelyn OoiOpen Connect Appliances - Jocelyn Ooi
Open Connect Appliances - Jocelyn Ooi
 
Traffic Engineering for CDNs
Traffic Engineering for CDNsTraffic Engineering for CDNs
Traffic Engineering for CDNs
 
Data Center Interconnectivity Trends and Their Effect on Optical Networking
Data Center Interconnectivity Trends and Their Effect on Optical NetworkingData Center Interconnectivity Trends and Their Effect on Optical Networking
Data Center Interconnectivity Trends and Their Effect on Optical Networking
 
1 of 18
Download to read offline
COMMERCIAL–IN-CONFIDENCECOMMERCIAL–IN-CONFIDENCE 100Gbps Core Network Deployment
COMMERCIAL–IN-CONFIDENCE Overall Network 2
COMMERCIAL–IN-CONFIDENCE Where Did The Upgrade Take Place? •  The upgrades were done in our Johannesburg and Cape Town PoP’s. •  South Africa is our fastest growing market. •  South Africa is a transit point for the WACS cable system. 3
COMMERCIAL–IN-CONFIDENCE Standard Core PoP Infrastructure 4 Cisco CRS Router 8-slot Chassis Cisco C6880-X Ethernet Switch
COMMERCIAL–IN-CONFIDENCE Why This Design? •  Core router ports are expensive. •  More efficient use of IP address space (LAN, not point-to-point links). •  The only traffic hitting the core routers is exiting the PoP. •  Hands-off the core routers. •  Easy to add edge devices within the PoP. •  Device upgrades can be detached/independent from/of each other. •  Avoid stranding bandwidth. 5
COMMERCIAL–IN-CONFIDENCE Previous Connectivity Compliment 6 14-port 10Gbps Line Card Cisco CRS-3 Router 140Gbps/slot 16-port 10Gbps Line Card C6880-X Switch 80Gbps/slot -  Each PoP was deployed with 80Gbps router ó switch in the core. -  Capacity provisioned as 2x 40Gbps LAG’s. -  Each LAG comprises 4x 10Gbps ports. -  Each router has links to each of the 2 switches. -  2 core routers + 2 core switches per PoP.
COMMERCIAL–IN-CONFIDENCE Previous Connectivity Compliment •  Generally, 80% of the core PoP’s are fine with this design. •  Johannesburg and Cape Town were starting to become a concern. •  Large EoMPLS flows that don’t load balance well. •  Transit traffic from other markets during a SEACOM cable outage. •  Overall network growth in Johannesburg and Cape Town. 7
COMMERCIAL–IN-CONFIDENCE New Connectivity Compliment 8 4-port 100Gbps Line Card Cisco CRS-X Router 400Gbps/slot 12-port 100Gbps Line Card Arista 7508E Switch 1.2Tbps/slot Arista 7508E Ethernet Switch 48-port 1Gbps/10Gbps Line Card Arista 7508E Switch 480Gbps/slot
COMMERCIAL–IN-CONFIDENCE Why Change To Arista? •  Cheapest 100Gbps/port core switch on the market (as at 2017). •  The Cisco Nexus 7710 was too expensive. •  The Juniper QFX10008 was too expensive. •  Arista supported all the basic features we need for a core switch: •  802.1Q. •  802.1AX (a.k.a 802.3ad). •  Load balancing both IP and non-IP traffic flows. 9
COMMERCIAL–IN-CONFIDENCE Why Keep The Cisco CRS? •  The Cisco CRS has been our standard core router since 2014. •  Options like the Cisco NCS and Juniper PTX were not available then. •  Nokia’s (ALU’s) 7950 XRS router would have been a new vendor. •  Although remote with each year, non-Ethernet requirements remain. •  The issue with the CRS: •  The 4-port 100Gbps line cards are too pricey. •  The CPAK optics are too costly. •  But no immediate demand needed in other PoP’s. •  Cisco claim maximum future per-slot throughput of 800Gbps. •  Engineering support in the future is “unknown”. •  Focus is on the NCS6000. •  But overall, the Cisco CRS is more than enough for our needs… 10
COMMERCIAL–IN-CONFIDENCE The Challenges •  Optics Incompatibility: •  Initial order was for the Nexus 7710 switch. •  The Cisco CPAK optics were based on the SR4 spec. •  The Arisa optics were based on the SR10 spec. •  We waited a month to get the right SR4 optics for the Arista. •  Poor 100Gbps experience in Africa: •  No one in the industry could assist with our challenges. •  Little to no experience in the African market. •  Lack of 100Gbps accessories in Africa: •  We were unable to source MPO cables locally in-continent. •  Our African suppliers had no knowledge of MPO cables. •  Resorted to our off-continent suppliers for the support. •  Some MPO cables shipped with the wrong termination. 11
COMMERCIAL–IN-CONFIDENCE The Challenges •  Logistics: •  100Gbps line cards are new to African Customs departments. •  Customs clearance of these was a major problem. •  Cisco CRS fabric upgrade: •  Pre-100Gbps setup was CRS-3, i.e., 140Gbps/slot. •  100Gbps required a fabric upgrade to CRS-X, i.e., 400Gbps/slot. •  This was known prior to the deployment. •  A change in in-house teams meant this was forgotten. •  Avenue of pleasure when the 100Gbps line card won’t boot J. •  Training on Arista: •  We have generally been a 2-vendor network. •  Cisco and Juniper only. •  Arista is new, so training was needed. •  But this wasn’t difficult. EOS looks like IOS. 12
COMMERCIAL–IN-CONFIDENCE The Challenges •  Vendor optics lock-in: •  Arista tried to force us to use their own optics. •  We insisted on using our own from 3rd party suppliers. •  Arista provided an unlock code to allow these on their platform. •  The cost savings for re-using the optics is over 95%. •  Only need to buy the 100Gbps optics. •  Data centre changes: •  The Cisco 6800-X used 10A power sources. •  The Arista 7508E needed 16A power sources. •  We also needed to obtain additional rack space for the Arista’s. 13
COMMERCIAL–IN-CONFIDENCE Final Result 14
COMMERCIAL–IN-CONFIDENCE Final Result 15
COMMERCIAL–IN-CONFIDENCE EOS vs. IOS •  In short, no real difference in the CLI. •  In fact, Cisco decided to sue Arista for “slavishly” copying IOS: http://bit.ly/eosvios •  Only need to learn about Arista-specific things. •  You can switch from Cisco to Arista in a matter of hours. 16
COMMERCIAL–IN-CONFIDENCE Too Many Vendors? •  SEACOM have always been a 2-vendor state: •  Cisco •  Juniper •  We did not see introducing Arista as a 3rd vendor as a problem: •  Very specific, simple version – Layer 2 Ethernet core switching. •  Low-touch requirements. 17
COMMERCIAL–IN-CONFIDENCE Thank You Q&A mark.tinka@seacom.mu 18

More Related Content

100Gbps Core Network Deployment in an African Network - Mark Tinka

  • 3. COMMERCIAL–IN-CONFIDENCE Where Did The Upgrade Take Place? •  The upgrades were done in our Johannesburg and Cape Town PoP’s. •  South Africa is our fastest growing market. •  South Africa is a transit point for the WACS cable system. 3
  • 4. COMMERCIAL–IN-CONFIDENCE Standard Core PoP Infrastructure 4 Cisco CRS Router 8-slot Chassis Cisco C6880-X Ethernet Switch
  • 5. COMMERCIAL–IN-CONFIDENCE Why This Design? •  Core router ports are expensive. •  More efficient use of IP address space (LAN, not point-to-point links). •  The only traffic hitting the core routers is exiting the PoP. •  Hands-off the core routers. •  Easy to add edge devices within the PoP. •  Device upgrades can be detached/independent from/of each other. •  Avoid stranding bandwidth. 5
  • 6. COMMERCIAL–IN-CONFIDENCE Previous Connectivity Compliment 6 14-port 10Gbps Line Card Cisco CRS-3 Router 140Gbps/slot 16-port 10Gbps Line Card C6880-X Switch 80Gbps/slot -  Each PoP was deployed with 80Gbps router ó switch in the core. -  Capacity provisioned as 2x 40Gbps LAG’s. -  Each LAG comprises 4x 10Gbps ports. -  Each router has links to each of the 2 switches. -  2 core routers + 2 core switches per PoP.
  • 7. COMMERCIAL–IN-CONFIDENCE Previous Connectivity Compliment •  Generally, 80% of the core PoP’s are fine with this design. •  Johannesburg and Cape Town were starting to become a concern. •  Large EoMPLS flows that don’t load balance well. •  Transit traffic from other markets during a SEACOM cable outage. •  Overall network growth in Johannesburg and Cape Town. 7
  • 8. COMMERCIAL–IN-CONFIDENCE New Connectivity Compliment 8 4-port 100Gbps Line Card Cisco CRS-X Router 400Gbps/slot 12-port 100Gbps Line Card Arista 7508E Switch 1.2Tbps/slot Arista 7508E Ethernet Switch 48-port 1Gbps/10Gbps Line Card Arista 7508E Switch 480Gbps/slot
  • 9. COMMERCIAL–IN-CONFIDENCE Why Change To Arista? •  Cheapest 100Gbps/port core switch on the market (as at 2017). •  The Cisco Nexus 7710 was too expensive. •  The Juniper QFX10008 was too expensive. •  Arista supported all the basic features we need for a core switch: •  802.1Q. •  802.1AX (a.k.a 802.3ad). •  Load balancing both IP and non-IP traffic flows. 9
  • 10. COMMERCIAL–IN-CONFIDENCE Why Keep The Cisco CRS? •  The Cisco CRS has been our standard core router since 2014. •  Options like the Cisco NCS and Juniper PTX were not available then. •  Nokia’s (ALU’s) 7950 XRS router would have been a new vendor. •  Although remote with each year, non-Ethernet requirements remain. •  The issue with the CRS: •  The 4-port 100Gbps line cards are too pricey. •  The CPAK optics are too costly. •  But no immediate demand needed in other PoP’s. •  Cisco claim maximum future per-slot throughput of 800Gbps. •  Engineering support in the future is “unknown”. •  Focus is on the NCS6000. •  But overall, the Cisco CRS is more than enough for our needs… 10
  • 11. COMMERCIAL–IN-CONFIDENCE The Challenges •  Optics Incompatibility: •  Initial order was for the Nexus 7710 switch. •  The Cisco CPAK optics were based on the SR4 spec. •  The Arisa optics were based on the SR10 spec. •  We waited a month to get the right SR4 optics for the Arista. •  Poor 100Gbps experience in Africa: •  No one in the industry could assist with our challenges. •  Little to no experience in the African market. •  Lack of 100Gbps accessories in Africa: •  We were unable to source MPO cables locally in-continent. •  Our African suppliers had no knowledge of MPO cables. •  Resorted to our off-continent suppliers for the support. •  Some MPO cables shipped with the wrong termination. 11
  • 12. COMMERCIAL–IN-CONFIDENCE The Challenges •  Logistics: •  100Gbps line cards are new to African Customs departments. •  Customs clearance of these was a major problem. •  Cisco CRS fabric upgrade: •  Pre-100Gbps setup was CRS-3, i.e., 140Gbps/slot. •  100Gbps required a fabric upgrade to CRS-X, i.e., 400Gbps/slot. •  This was known prior to the deployment. •  A change in in-house teams meant this was forgotten. •  Avenue of pleasure when the 100Gbps line card won’t boot J. •  Training on Arista: •  We have generally been a 2-vendor network. •  Cisco and Juniper only. •  Arista is new, so training was needed. •  But this wasn’t difficult. EOS looks like IOS. 12
  • 13. COMMERCIAL–IN-CONFIDENCE The Challenges •  Vendor optics lock-in: •  Arista tried to force us to use their own optics. •  We insisted on using our own from 3rd party suppliers. •  Arista provided an unlock code to allow these on their platform. •  The cost savings for re-using the optics is over 95%. •  Only need to buy the 100Gbps optics. •  Data centre changes: •  The Cisco 6800-X used 10A power sources. •  The Arista 7508E needed 16A power sources. •  We also needed to obtain additional rack space for the Arista’s. 13
  • 16. COMMERCIAL–IN-CONFIDENCE EOS vs. IOS •  In short, no real difference in the CLI. •  In fact, Cisco decided to sue Arista for “slavishly” copying IOS: http://bit.ly/eosvios •  Only need to learn about Arista-specific things. •  You can switch from Cisco to Arista in a matter of hours. 16
  • 17. COMMERCIAL–IN-CONFIDENCE Too Many Vendors? •  SEACOM have always been a 2-vendor state: •  Cisco •  Juniper •  We did not see introducing Arista as a 3rd vendor as a problem: •  Very specific, simple version – Layer 2 Ethernet core switching. •  Low-touch requirements. 17