SlideShare a Scribd company logo

DevCon13 System Administration Basics

•
•
S
sysnickm

This document provides an overview of system administration basics including hardware, virtualization, networking, load balancing, databases, monitoring, integration, and vendor management. It discusses topics such as choosing hardware vendors, datacenter requirements, hardware sizing, redundancy, virtualization platforms, application and database server virtualization, networking components like firewalls and switches, load balancing options, database best practices around backups, compression, and indexes, monitoring tools, integration, and managing relationships with software vendors and support.

1 of 45
System Administration Basics Nick McClure Operating Systems Programmer III University of Kentucky
ABOUT ME Nick McClure Operating Systems Programmer III University of Kentucky nickjm@uky.edu @sysnickm
WHAT WE ARE GOING TO LEARN TODAY • Hardware • Virtualization • Network • Load Balancing • Database • Monitoring • Integration • Vendor Management
THE HARDWARE Pick your vendor • Intel processors and chipsets are the same across hardware vendors • Virtualization layer abstracts hardware anyway • OS developers work closely with most hardware manufactures to ensure compatibility
DATACENTER Ensure your facility can handle the hardware • Provide enough power • Provide enough cooling • Control access to facility • Keep it simple
HARDWARE SIZING • Difficult to size a deployment without usage metrics • Difficult to get accurate metrics without a well sized solution • Start with vendor best practices and community experience • Every institution will have a different pattern • If in a shared environment, know what you are shared with
REDUNDANCY Institution Administration sets the goal • IT creates plan to achieve it • Redundancy can be expensive • Create plans based on likely events, describe what is needed to mitigate risk
VIRTUALIZATION • Modern Hardware is designed around virtualization • Intel works directly with hypervisor vendors • Most modern apps run from virtualized servers
APPLICATION SERVER VIRTUALIZATION • If you aren’t virtualized here, you should be • Most of our workloads are memory heavy • Memory is cheap • Processor vendors have made the CPU easy to share • Hypervisors have advanced features for performance management • VMotion • Live Migration
DATABASE SERVER VIRTUALIZATION • Different Ways to virtualize database servers • Hyper-Visor – Depends on requirements – Large virtualization environments can host large databases • Multi-Instance Cluster – Allows for high performance databases in a shared environment • Database System can be virtualized effectively • Understand your environment
HYPERVISOR TAX • VMware is a great product • VMware license costs more than the hardware • Annual support costs • If you aren’t going to use the VMware feature set, try something else • Linux KVM • Microsoft Hyper-V
SOFTWARE LICENSES • Software Licensing can be different when virtualized • Talk to your software vendor • Oracle and SQL Server have special considerations with running virtually
THE NETWORK When something goes wrong everybody blames the network
SCOPE Physical or Virtual infrastructure for packet transmission • Switches • Firewalls • Cabling
BASIC NETWORK
FIREWALLS • Traditional firewalls provide good protection • Block all, allow only the minimum required • Audit rules regularly • Limit surface area based on system type
MULTIPLE FIREWALLS Place a firewall between the Load Balancer and the App servers • Load Balancer is generally exposed to the public internet • Protects App servers from a compromised Load Balancer
MULTIPLE FIREWALLS Place a firewall between your app servers and data servers • Web Applications servers expose more surface area to the internet, even with a load balancer • Compromised app servers could allow an attacker to touch the database virtually unnoticed
FIREWALL LATENCY • Searching the Access Control List (ACL) can be an expensive operation • Load Balancers maintain active connections with application servers • Application servers maintain active connections with database and file servers • ACL search usually only happens at app start up
SWITCHES • 10Gb is now mainstream • Most servers are available with 10Gb NICs on board • Virtual Environments should be based on at least 10Gb
LOAD BALANCING • Dedicated Hardware Load Balancers • Virtual Load Balancers • DNS Round Robin
DEDICATED HARDWARE • Offers Best performance and Options • Custom hardware to hand encryption, compression, etc. • Can provide intelligent balancing rules based on real time monitoring • Expensive
VIRTUAL LOAD BALANCER • Offers good performance • Works with your existing hypervisor platform • Low Overhead • Can provide intelligent balancing rules based on real time monitoring • No custom hardware • Less expensive
DNS ROUND ROBIN • Offers a minimum level of protection • No integrated monitoring • Scripts can monitor and adjust DNS • DNS TTL can still cause issues
LOAD BALANCING Modern Applications work better when the load is spread across multiple systems • Load balancer can direct traffic to servers best suited to handle load • Can stop sending traffic to failed, or slow servers
SSL OFFLOADING Many hardware load balancers offer SSL Acceleration • Blackboard currently supports SSL Offloading in two modes • SSL Re-Encrypt • SSL Offloading • Both methods allow for cookie insertion for session affinity
SSL OFFLOADING Offloading SSL provides some benefits • Latency • Cuts down on processing time by both application server and load balancer • Licensing • Many load balancers license SSL based on Transaction per Second • Re-Encrypt doubles your license requirement
OFFLOADING AND SECURITY If you don’t trust your network, you shouldn’t put your application servers there. • RDBMS Transaction not encrypted • NFS/SMB not encrypted
DATABASE • Blackboard relies heavily on the database, even when idle • Follow best practices from the vendor • If its working, don’t touch it unless you’ve tested
BACKUPS • Full Database Backups • Regular Backups to an enterprise backup solution • Log Backups • Run between full backups to catch changes • Truncates the active transaction log • Run more frequently • Every 4 hours is good • Hourly is better
COMPRESSION Current versions of SQL Server and Oracle support data compression • Disk IO is a common bottleneck • Disk IO is more expensive than CPU time • Saw an 80% reduction of disk space on many objects when compressed • Performance improved with compression
INDEXES • Indexes can greatly improve performance • Bad indexes can impact performance • Reorganize indexes regularly • Rebuild indexes when needed • Some versions of SQL Server require index be offline to rebuild
THE FUTURE Microsoft SQL Server 2014 supports in- memory OLTP • Potential for dramatic performance improvement • No extra cost
MONITORING • Managers Demand Metrics • Blackboard is Mission Critical • It isn’t about IF, it is about WHEN
TOOLS There are several tools available • Nagios • SCOM • OpenView • SolarWinds • Many more
SYNTHETIC TRANSACTIONS • Allow the system to perform actions like a user • Determine subsystem availability • Can give metrics on subsystem performance
PUBLISH METRICS By publishing your metrics, you can control the message • Blog • Email • Twitter • Staff meeting
ADVANCED MONITORING Several options for deeper monitoring • End user experience monitoring • JVM monitoring • Database engine monitoring • Tools can become very expensive
INTEGRATION • Data from SIS/ERP should be automated • Hand entered data can result in errors • Blackboard provides several integration options • Several Sessions at DevCon and BbWorld
VENDOR MANAGEMENT • Don’t make assumptions
SUPPORT CASES Provide as much information in the initial case notes as you can • Provide screen shots • Provide a video capture of the issue if you can
SUPPORT CASE TEMPLATE • URL • Authentication info • Blackboard version, including patches • Server OS version • Database server version • Number of courses and/or users affected • Odd configuration options • Better to bring it up first • Support will usually work with your on these • Instance (Production, Test, etc.)
SUPPORT MANAGERS • Support managers help prioritize things • Can assist with ticket escalation • Monitor community discussions and social networks
ACCOUNT REPRESENTATIVE • Your account rep is your advocate • Their job is to keep you happy with Blackboard • Can assist with case escalation • Can assist with feature requests
THANK YOU! Nick McClure Operating Systems Programmer III University of Kentucky nickjm@uky.edu @sysnickm If you would like to provide feedback for this session please email: BbWorldFeedback@blackboard.com The title of this session is: System Administrator Basics

More Related Content

DevCon13 System Administration Basics

  • 1. System Administration Basics Nick McClure Operating Systems Programmer III University of Kentucky
  • 2. ABOUT ME Nick McClure Operating Systems Programmer III University of Kentucky nickjm@uky.edu @sysnickm
  • 3. WHAT WE ARE GOING TO LEARN TODAY • Hardware • Virtualization • Network • Load Balancing • Database • Monitoring • Integration • Vendor Management
  • 4. THE HARDWARE Pick your vendor • Intel processors and chipsets are the same across hardware vendors • Virtualization layer abstracts hardware anyway • OS developers work closely with most hardware manufactures to ensure compatibility
  • 5. DATACENTER Ensure your facility can handle the hardware • Provide enough power • Provide enough cooling • Control access to facility • Keep it simple
  • 6. HARDWARE SIZING • Difficult to size a deployment without usage metrics • Difficult to get accurate metrics without a well sized solution • Start with vendor best practices and community experience • Every institution will have a different pattern • If in a shared environment, know what you are shared with
  • 7. REDUNDANCY Institution Administration sets the goal • IT creates plan to achieve it • Redundancy can be expensive • Create plans based on likely events, describe what is needed to mitigate risk
  • 8. VIRTUALIZATION • Modern Hardware is designed around virtualization • Intel works directly with hypervisor vendors • Most modern apps run from virtualized servers
  • 9. APPLICATION SERVER VIRTUALIZATION • If you aren’t virtualized here, you should be • Most of our workloads are memory heavy • Memory is cheap • Processor vendors have made the CPU easy to share • Hypervisors have advanced features for performance management • VMotion • Live Migration
  • 10. DATABASE SERVER VIRTUALIZATION • Different Ways to virtualize database servers • Hyper-Visor – Depends on requirements – Large virtualization environments can host large databases • Multi-Instance Cluster – Allows for high performance databases in a shared environment • Database System can be virtualized effectively • Understand your environment
  • 11. HYPERVISOR TAX • VMware is a great product • VMware license costs more than the hardware • Annual support costs • If you aren’t going to use the VMware feature set, try something else • Linux KVM • Microsoft Hyper-V
  • 12. SOFTWARE LICENSES • Software Licensing can be different when virtualized • Talk to your software vendor • Oracle and SQL Server have special considerations with running virtually
  • 13. THE NETWORK When something goes wrong everybody blames the network
  • 14. SCOPE Physical or Virtual infrastructure for packet transmission • Switches • Firewalls • Cabling
  • 16. FIREWALLS • Traditional firewalls provide good protection • Block all, allow only the minimum required • Audit rules regularly • Limit surface area based on system type
  • 17. MULTIPLE FIREWALLS Place a firewall between the Load Balancer and the App servers • Load Balancer is generally exposed to the public internet • Protects App servers from a compromised Load Balancer
  • 18. MULTIPLE FIREWALLS Place a firewall between your app servers and data servers • Web Applications servers expose more surface area to the internet, even with a load balancer • Compromised app servers could allow an attacker to touch the database virtually unnoticed
  • 19. FIREWALL LATENCY • Searching the Access Control List (ACL) can be an expensive operation • Load Balancers maintain active connections with application servers • Application servers maintain active connections with database and file servers • ACL search usually only happens at app start up
  • 20. SWITCHES • 10Gb is now mainstream • Most servers are available with 10Gb NICs on board • Virtual Environments should be based on at least 10Gb
  • 21. LOAD BALANCING • Dedicated Hardware Load Balancers • Virtual Load Balancers • DNS Round Robin
  • 22. DEDICATED HARDWARE • Offers Best performance and Options • Custom hardware to hand encryption, compression, etc. • Can provide intelligent balancing rules based on real time monitoring • Expensive
  • 23. VIRTUAL LOAD BALANCER • Offers good performance • Works with your existing hypervisor platform • Low Overhead • Can provide intelligent balancing rules based on real time monitoring • No custom hardware • Less expensive
  • 24. DNS ROUND ROBIN • Offers a minimum level of protection • No integrated monitoring • Scripts can monitor and adjust DNS • DNS TTL can still cause issues
  • 25. LOAD BALANCING Modern Applications work better when the load is spread across multiple systems • Load balancer can direct traffic to servers best suited to handle load • Can stop sending traffic to failed, or slow servers
  • 26. SSL OFFLOADING Many hardware load balancers offer SSL Acceleration • Blackboard currently supports SSL Offloading in two modes • SSL Re-Encrypt • SSL Offloading • Both methods allow for cookie insertion for session affinity
  • 27. SSL OFFLOADING Offloading SSL provides some benefits • Latency • Cuts down on processing time by both application server and load balancer • Licensing • Many load balancers license SSL based on Transaction per Second • Re-Encrypt doubles your license requirement
  • 28. OFFLOADING AND SECURITY If you don’t trust your network, you shouldn’t put your application servers there. • RDBMS Transaction not encrypted • NFS/SMB not encrypted
  • 29. DATABASE • Blackboard relies heavily on the database, even when idle • Follow best practices from the vendor • If its working, don’t touch it unless you’ve tested
  • 30. BACKUPS • Full Database Backups • Regular Backups to an enterprise backup solution • Log Backups • Run between full backups to catch changes • Truncates the active transaction log • Run more frequently • Every 4 hours is good • Hourly is better
  • 31. COMPRESSION Current versions of SQL Server and Oracle support data compression • Disk IO is a common bottleneck • Disk IO is more expensive than CPU time • Saw an 80% reduction of disk space on many objects when compressed • Performance improved with compression
  • 32. INDEXES • Indexes can greatly improve performance • Bad indexes can impact performance • Reorganize indexes regularly • Rebuild indexes when needed • Some versions of SQL Server require index be offline to rebuild
  • 33. THE FUTURE Microsoft SQL Server 2014 supports in- memory OLTP • Potential for dramatic performance improvement • No extra cost
  • 34. MONITORING • Managers Demand Metrics • Blackboard is Mission Critical • It isn’t about IF, it is about WHEN
  • 35. TOOLS There are several tools available • Nagios • SCOM • OpenView • SolarWinds • Many more
  • 36. SYNTHETIC TRANSACTIONS • Allow the system to perform actions like a user • Determine subsystem availability • Can give metrics on subsystem performance
  • 37. PUBLISH METRICS By publishing your metrics, you can control the message • Blog • Email • Twitter • Staff meeting
  • 38. ADVANCED MONITORING Several options for deeper monitoring • End user experience monitoring • JVM monitoring • Database engine monitoring • Tools can become very expensive
  • 39. INTEGRATION • Data from SIS/ERP should be automated • Hand entered data can result in errors • Blackboard provides several integration options • Several Sessions at DevCon and BbWorld
  • 40. VENDOR MANAGEMENT • Don’t make assumptions
  • 41. SUPPORT CASES Provide as much information in the initial case notes as you can • Provide screen shots • Provide a video capture of the issue if you can
  • 42. SUPPORT CASE TEMPLATE • URL • Authentication info • Blackboard version, including patches • Server OS version • Database server version • Number of courses and/or users affected • Odd configuration options • Better to bring it up first • Support will usually work with your on these • Instance (Production, Test, etc.)
  • 43. SUPPORT MANAGERS • Support managers help prioritize things • Can assist with ticket escalation • Monitor community discussions and social networks
  • 44. ACCOUNT REPRESENTATIVE • Your account rep is your advocate • Their job is to keep you happy with Blackboard • Can assist with case escalation • Can assist with feature requests
  • 45. THANK YOU! Nick McClure Operating Systems Programmer III University of Kentucky nickjm@uky.edu @sysnickm If you would like to provide feedback for this session please email: BbWorldFeedback@blackboard.com The title of this session is: System Administrator Basics

Editor's Notes

  1. Don’t want your database server shared with video encoding
  2. Some software vendors won’t license their software for virtualization at all
  3. Some users will see failures until manual intervention
  4. Compression varies based on table and index values
  5. Evaluate them, Pick one, and Use it
  6. Based on your needs these tools can be valuable for Troubleshooting and RCA