Provisioning Servers Made Easy
•
1 like•499 views
This document discusses Linux server provisioning using Stacki. Stacki is a tool that automates the provisioning of Linux servers at scale from bare metal to a fully configured system. It addresses the exponential complexity of managing large clusters as more servers are added. Stacki handles all aspects of server provisioning from OS installation to configuration of networking, storage, software and more. It provides a fully automated, repeatable process to quickly deploy and manage servers.
Report
Share
Provisioning Servers Made Easy
- 1. Provisioning Linux Servers Made Easy Greg Bruno, PhD VP Engineering, StackIQ
- 4. Open Source Stack Installer Stacki is a very fast and ultra reliable Linux server provisioning tool … at scale. With zero prerequisites for taking systems from bare metal to a ping and prompt.
- 5. Why is this hard and important?
- 6. Datacenter Architecture Frontend Network Backend Backend Backend Backend em1 em1 em1 em1 em1
- 7. Datacenter Host Software Stack DevOps / Configuration Tool DHCP / DNS / TFTP NetworkDiskOS In-house developed deployment tools - Disk Array Controller Configuration - Disk Partitioning Configuration
- 8. The “Step 0” Problem Check namenodes are empty Format/start HDFS Create all directories Create all metastores Start services (Hbase, Hive, Oozie, Sqoop, Impala, etc) Deploy client configuration Configure database Setup/assign monitors (activity, services, and host) Test database connections Validate/resolve hostnamesConsistent host timezones No bad kernel versions running (CDH) version consistency Java version consistency Daemons versions consistency Mgmt Agents versions consistency Host specification/SSH ports MUCH MORE … DHCP Server/Client setup TFTP/PXE configuration Server OS installation Node OS Install RAID configuration Boot configuration System/data disk partitioning Monitoring system setup and config Lights Out/IPMI setup User accounts added and synced SSH keys on all hosts Network node configuration Config Mgmt install and configuration Route configurationOS upgrades/updates Site specific software and configuration Host specification/SSH ports Security Firewall setupCluster Mgmt utility Database install and config Multiple network configPackage installation MUCH MORE …
- 9. Clusters are Different Adding new servers does require coordination Newly added servers must: • Have same software stack as original servers • Have same configuration as original servers • Know about original servers And, original servers must: • Know about new servers Result: The management complexity added to the Operations staff is “exponential”
- 10. Exponential Complexity Number of Servers ManagementComplexity General Data Center Clusters
- 11. The Pain Curve Number of Servers ManagementComplexity General Data Center Clusters PAIN
- 12. The Pain Threshold The pain threshold differs for every organization Function of: • cluster(s) size • number of people in Operations • Operations staff cluster expertise
- 13. Moore’s Law 50 1 2 3 4 8 1 2 3 4 5 6 7 Time (Years) Density 18 month doubling
- 14. Moore’s Law and Infrastructure Value
- 15. What it Means for You 50 1 2 3 4 100 0 10 20 30 40 50 60 70 80 90 Time (Years) Value(%) 3 months 90% value 18 months 50% value
- 16. Time is Money The clock starts ticking when hosts land on your loading dock Without your applications online, you have an paper weight that consumes power, cooling, and management’s attention
- 17. How We Solve the Problem
- 18. History • San Diego Supercomputer Center • 1986 - National Science Foundation • Along with NCSA only two non-classified centers • Mission: serve computational scientists • Rocks • 2000 - First cluster group inside SDSC • Version 1.0 released that November as open source • 10k+ clusters world-wide • StackIQ • 2006 - Commercial support for Rocks • 2011 - Venture Backed • Focus on next generation clustered systems (Data, Cloud) • Stacki - 2015 • June – released as open source • July – first hyper-scale user
- 19. Philosophy Make it – Automatic ◦ Think about it, test it. Deploy it. ◦ People don’t scale, software does. Free your people – allow ops guys to be ops/analysis guys, move them from single machine view to global machine view. Make it – Repeatable ◦ State of the environment is guaranteed. Does not require homogeneity of hardware or functionality. Make compute environments homogenous on heterogeneous hardware and software. ◦ Really, nothing is homogenous. Environment maybe, behavior of that environment on different machines while predictable will not be the same across all hardware. Stacki gets you flexibility and predictability. Make it – Reliable ◦ You always get what you want when you want it. You can make reasonable estimates of need because you’ve made the environment predictable and repeatable. Just like science! Make it – Comprehensive ◦ Manage application layer(s) down to kernels and device configuration with one tool. Never hit the network unconfigured. ◦ Provide turn-key deployment with reasonable default settings and ability to customize / re-wire as desired.
- 20. Stacki Positioning DevOps / Configuration Tool DHCP / DNS / TFTP NetworkDiskOS In-house developed deployment tools - Disk Array Controller Configuration - Disk Partitioning Configuration
- 21. Datacenter Architecture Frontend Network Backend Backend Backend Backend em1 em1 em1 em1 em1
- 22. Download and Boot the ISO Go to www.stacki.com and download the ISO ◦ It’s 1.8 GB ◦ “stacki” pallet plus stripped down CentOS 6.6 Boot the ISO on the host that will be your frontend
- 29. Frontend Services Services to build backend nodes ◦ DHCP ◦ TFTP ◦ Named (optional) Services to access backend nodes ◦ SSH key management ◦ Parallel execution shell
- 31. Frontend Network Backend Backend Backend Backend em1 em1 em1 em1 em1 Backend Installation Save your Host Configuration spreadsheet as a CSV Import CSV on frontend ◦ “stack load hostfile file=hosts.csv” Tell backend nodes to install on their next PXE boot ◦ “stack set host boot backend action=install” PXE boot all backend nodes Done!
- 34. Advanced Networking Via Host Configuration spreadsheet, you can configure: ◦ Bonded interfaces ◦ VLANs ◦ Bridging ◦ Any combo of the above Manage hosts in multiple subnets ◦ Build a single cluster from hosts in multiple subnets ◦ Manage hosts in multiple datacenters
- 36. Disk Controller Configuration Spreadsheet
- 37. Disk Partition Configuration Spreadsheet
- 38. Multiple Distributions A frontend houses a default distribution ◦ Based on stripped down CentOS 6.6 or 7.1 ◦ Used to build backend nodes Can add any number of new distributions to a frontend ◦ E.g., RHEL 6.x based distro, CentOS 6.5, etc. Assign any backend node to any distro
- 39. PayPal
- 40. Hadoop @ PayPal 12 x 2TB SATA data drives 48 nodes each rack 1GBE-10GBE NICs 24 x 900GB 6G SAS 10K data drives 24 nodes each rack 10GBE NIC 8 x 4TB NR-SAS data drives 10 GBE NIC Bay Area Salt Lake City Las Vegas DATACENTERS • 3,000 nodes and growing • 60+ initial server racks • Heterogeneous HW across multiple DCs Data Science Infrastructure Footprint 48 nodes each rack
- 41. Automation Challenge Spinout creates some datacenter automation challenges … • Smaller team but even more to do • Rethink automation • Distributed systems have tons of local drives which require time consuming disk formatting and partitioning, and hardware RAID config on masternodes • New provisioning solution needs to easily, flexibly integrate w/ other commercial, open source, and homegrown management tools • Can 100s or 1000s of nodes be (re)provisioned as quickly as one or a few? (e.g., drive failures mean replacing entire host from O/S to disk to network to firmware to … etc)
- 42. Stacki @ PayPal Ambari HDP Health Detection Integration IPMI/iLOOS Disk Network DHCP / DNS / TFTP Ansible - Disk Array Controller Configuration - Disk Partitioning Configuration “Stacki + Ansible = Happiness. :D” – Stacki mailing list 8/11/15
- 43. Quick, Early Success 14 Minutes*To Fully Provision 6 Racks of Bare Metal (288 Servers) Includes wiping all disks then fully partitioning & formatting ~3500 drives And Now… Upgrades all firmware automatically Executes Ansible scripts on all hosts Hadoop packages installed * Versus hours with other hyperscale management tools, or days to weeks with traditional tools and processes
- 44. Try It Out
- 45. stacki.com Download - www.stacki.com Source & Docs - github.com/StackIQ/stacki/wiki Discuss - groups.google.com/forum/#!forum/stacki
- 46. PayPal’s Options Bring what we used at former parent company eBay with us. Build our own soups-to-nuts bespoke bare metal provisioning tool. Find the perfect open source tool that we can use and grow with. Not Possible Not Optimal Not Likely
- 47. Quick, Early Success 2 Weeks Instead of 2 Years To Build a Scale-out Management Solution 1. Installed Stacki Frontend (base management server) Ran test installations of backend servers 1. Single Server test 2. Full Rack test (48 nodes) 2. Updated distribution (CentOS 6.6) to install additional packages 3. Integrated IPMI information into Stacki 1. Can now ssh into all IPMI consoles from the Stacki frontend host using <hostname>.ipmi 4. Re-ran with PayPal kickstart changes/additions and was able to image 6 racks in 14 minutes, including: 1. Nuking disks/partitions and running a full format of all data drives 5. Updated the Stacki post-boot piece to do the following: 1. Upgrade firmware if host needs it 2. Runs PayPal Ansible playbook, which: 1. Installs additional packages 2. Creates user accounts 3. Disables unused services 4. Sets up resolver/ntp/syslog-ng/sudoers/limits. d/sysctl/etc. 5. Installs/configures Ambari agents 6. Checks data drive mounts, fstab 7. Prepares the rack to be added to a Hadoop cluster PayPal development with Stacki includes:
- 48. DevOps Agnostic DevOps / Configuration Tool DHCP / DNS / TFTP NetworkDiskOS In-house developed deployment tools - Disk Array Controller Configuration - Disk Partitioning Configuration
- 49. The “Step 0” Problem Check namenodes are empty Format/start HDFS Create all directories Create all metastores Start services (Hbase, Hive, Oozie, Sqoop, Impala, etc) Deploy client configuration Configure database Setup/assign monitors (activity, services, and host) Test database connections Validate/resolve hostnamesConsistent host timezones No bad kernel versions running (CDH) version consistency Java version consistency Daemons versions consistency Mgmt Agents versions consistency Host specification/SSH ports MUCH MORE … DHCP Server/Client setup TFTP/PXE configuration Server OS installation Node OS Install RAID configuration Boot configuration System/data disk partitioning Monitoring system setup and config Lights Out/IPMI setup User accounts added and synced SSH keys on all hosts Network node configuration Config Mgmt install and configuration Route configurationOS upgrades/updates Site specific software and configuration Host specification/SSH ports Security Firewall setupCluster Mgmt utility Database install and config Multiple network configPackage installation MUCH MORE … App Config Site Config HW Install System Performance Validation Bare Metal Installers Hadoop Mgmt Tool Upgrades/Patching Disk Configuration Monitoring Tool Configuration Tool Network/Site Config ToolsSystems Mgmt Tool Others … MANUAL SEMI-AUTOMATED TOOLCHAIN (w/o StackIQ) w/StackIQ FULLY AUTOMATED
- 50. StackIQ Boss
- 51. Configuration Database Server appliance types (e.g. data, namenode, tomcat, …) Number of CPUs Disk partitioning Hardware RAID config PCI bus information … And other System Attributes
- 52. Attributes Global ◦ stack set attr Appliance ◦ stack set appliance attr OS ◦ stack set os attr Host ◦ stack set host attr
- 54. Zoom In
- 55. Starting from the Empty Set { }
- 56. { os } © 2009 UC Regents
- 57. { os, core } © 2009 UC Regents
- 58. { os, core, kernel } © 2009 UC Regents
- 59. { os, core, kernel, mapr } © 2009 UC Regents
- 60. Manage the Deltas {os, core, kernel, mapr} {os, core, kernel, horton} © 2009 UC Regents