High Availability Perl DBI + MySQL
•Download as PPT, PDF•
2 likes•2,348 views
A talk I gave @ YAPC::NA in 2005 on implementing High Availability for Perl applications that span multiple datacenters.
Report
Share
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (12)
Similar to High Availability Perl DBI + MySQL
Similar to High Availability Perl DBI + MySQL (20)
More from Steve Purkis
More from Steve Purkis (15)
Recently uploaded
Recently uploaded (20)
High Availability Perl DBI + MySQL
- 1. High Availability? Setting Up MySQL Our Approach DBI Framework Questions High Availability DBI & MySQL YAPC::NA 2005 High Availability DBI & MySQL Steve Purkis
- 2. High Availability? Setting Up MySQL Our Approach DBI Framework Questions High Availability DBI & MySQL
- 3. Slide 3 High Availability? • What is High Availability? • Ever have an application fail because a mission critical server crashed? • High Availability is an attempt to provide zero application downtime.
- 4. Slide 4 High Availability? • How do I achieve HA?: • Analyze your apps • Most important components? • Remove single points of failure (SPOFs): • Add redundant servers • Replicate data (backups, realtime) • Load balancing • Failover when things go pear-shaped
- 5. Slide 5 High Availability? • This talk… • Introduces one approach to building HA Perl apps that rely on MySQL. • We’ve used similar techniques with Oracle • They work for us… • They may not be right for you!
- 6. High Availability? Setting Up MySQL Our Approach DBI Framework Questions High Availability DBI & MySQL
- 7. Slide 7 Setting Up MySQL • Some typical architectures…
- 8. Slide 8 Setting Up MySQL • Ye olde basic setup • Not HA. Application Server MySQL Server
- 9. Slide 9 Setting Up MySQL • One master, one read-only slave • Good for load balancing • Still not really HA! Application Server MySQL Master MySQL Slave Data Replication reads writes Load Balancer
- 10. Slide 10 Setting Up MySQL • Two masters • It’s HA, but… • Flaky, tricky to work with! Application Server MySQL MasterTwo-way Replication MySQL Master Load Balancer
- 11. Slide 11 MySQL Slave Setting Up MySQL • Two masters, multiple slaves • Only one master active at a time • HA Application Server MySQL Slaves Data Replication reads writes Primary Master 2nd ary Master Master Failover Load Balancer • HA (unless your hosting center goes down)
- 12. Slide 12 Setting Up MySQL • Your HA MySQL architecture will depend on: • SLAs • Application requirements • Load • Physical spread of servers • Budget • Caveats: • Are your apps read or write heavy? • Beware of replication lag time • Common in high-latency networks • Do your apps need the latest data? • Transactions?
- 13. Slide 13 Setting Up MySQL • Load balancing • Hardware? • Software? • Master Failover • On Slaves: CHANGE MASTER TO … • Manual? • Automate? • Cron? Application? SLB? • On App servers: • Connect to Secondary • Not required if using a hot spare w/same ip
- 14. Slide 14 Setting Up MySQL • Recommended reading: • High Performance MySQL - Jeremey Zawodny & Derek Balling • MySQL Replication docs • MySQL Cluster white paper • For Linux: • Linux Virtual Server Project • Linux-HA
- 15. Slide 15 Setting Up MySQL • If you have any trouble: • MySQL mailing list
- 16. High Availability? Setting Up MySQL Our Approach DBI Framework Questions High Availability DBI & MySQL
- 17. Slide 17 Our Approach • Our requirements: • Read-heavy apps • Over 750 clients, many with SLAs • Reliability: • Read availability most important • Must work if a server farm goes down • Speed is of essence: • Millions of requests a day • Don’t need transactions • Minimize costs
- 18. Slide 18 Our Approach Application Server Application Server Application Server MySQL Slave MySQL Slaves MySQL Slaves Application Server MySQL Master MySQL 2nd ary MasterFailover Data Replication 95% 5% reads writes
- 19. Slide 19 Farm 2 Farm 3 Farm 1 Our Approach Application Server Application Server MySQL Slave Application Server MySQL Master MySQL 2nd ary MasterFailover In-Farm Data Replication Application Server MySQL Slave MySQL Slaves MySQL Slave MySQL Slaves Query caching
- 20. Slide 20 Our Approach • DBI Framework: • Load balancing • Server selection • Read / Write query? • Failover Application DBI Framework ----------------------------------- snip: how to tell reads from writes? --------------------------------- my ($action, $dbh) = $query =~ /Aselect|A(select|Ashow|Adesc/i ? ("read", $dbh_read) : ("write", $dbh_write); ----------------------------------------------------------------------------------------------------------
- 21. Slide 21 Our Approach • Load balancing • On connect • Weighted server selection • Availability, number of processes, user weights, which farm • MySQL idle timeout • Failover • Connect to next slave • Automatic query retry Application DBI Framework MySQL Slave MySQL Slave reads
- 22. Slide 22 Our Approach • Failover • Persistent connection to all masters • Automatic query retry • Automatic fallback Application Server DBI Framework Primary Master 2nd ary Master MasterFailover writes
- 23. High Availability? Setting Up MySQL Our Approach DBI Framework Questions High Availability DBI & MySQL
- 24. Slide 24 DBI Framework • Two wrapper functions: • dbconnect( 'read' | 'write' ) • Read: Select slave to connect to • Write: Connect to all masters • $sth = sql( $query ) • Read / write dbh selection • Failover & fallback • Pseudo code?
- 25. Slide 25 DBI Framework • dbconnect( 'read' | 'write' ) • Read: For each slave weight = 0, next if can’t ping check number of processes with mysqladmin weight = no. processes / user weighting Connect to slave with lowest weight Sanity check: run a simple query Try next slave if that failed • Write: For each master Connect Sanity check: run a simple query Set $write_dbh to primary master
- 26. Slide 26 DBI Framework • $sth = sql( $query ) • Determine query type… • Read: Execute query. Failover to next slave on error, and retry query. • Write: Fallback? If not using primary, and we failed over X seconds ago, try reconnecting to master. Execute query. Failover to next master on error, and retry query.
- 27. Slide 27 DBI Framework • Major Drawbacks: • Using DBI-based CPAN modules is hard!
- 28. Slide 28 DBI Framework • Looking Ahead… • Push HA logic into the DBI layer • Write DBD::MysqlHA ? • DBIx::HA ? • DBD::Multiplex ? • DBIx::DBCluster ? • SQL Relay ? • MySQL Cluster?
- 29. High Availability? Setting Up MySQL Our Approach DBI Framework Questions High Availability DBI & MySQL
- 30. Slide 30 DBI Frameworks on CPAN • If there’s time…
- 31. Slide 31 DBI Frameworks on CPAN • The ones I know a bit about: • DBIx::HA • DBD::Multiplex • DBIx::DBCluster
- 32. Slide 32 DBI Frameworks on CPAN • DBIx::HA • Generic HA solution (written & tested for Sybase) • Configurable by db name (%DATABASE::conf) • Looks well thought out
- 33. Slide 33 DBIx::HA Pros • Does Failover • On query failure & dbh disconnected • Does timeouts: • connect, query execute • Safe signals • Connect all dsns on init • Supports Apache::DBI Cons • No read / write distinction • No way to choose which dbh to use on failover • (want to use a mysql- specific algorithm) • Timeouts with SIGALRM • (but how else, really?) • No ping checks for non- Apache::DBI • (uses $dbh->ping anyways - we prefer ICMP ping)
- 34. Slide 34 DBIx::HA • Also: written & tested for Sybase! • Potential DBD::Mysql problems: • auto_reconnect we may reconnect to a db when we should be failing over
- 35. Slide 35 DBI Frameworks on CPAN • DBD::Multiplex • send requests to multiple dsn's • Configure servers to use in $dsn (pipe-separated list)
- 36. Slide 36 DBD::Multiplex Pros • Supports master/slave setup: • differentiates between reads / writes • Connects to all dsns (good for fast master failover) • Does failover: • default behaviour reads: first_success writes: first_error Cons • Can only specify one master • (though if you specify none, writes can go to all with 'first_error' exit mode) • Connects to all dsns • (don't want to connect to all slaves) • No customizable slave failover algorithm • No reconnects • No fallback!
- 37. Slide 37 DBI Frameworks on CPAN • How could we re-use CPAN modules? • DBIx::HA • Sub-class to introduce MySQL specific functionality? • Introduce a callback for server selection on failover? • Use in conjunction with DBD::Multiplex for read/write dbh selection? • There could be problems though… • In general: • Backwards compat • sql() wrapper (for backwards compat) • Custom logging? • Our db wrappers do other things too…
- 38. Slide 38 DBI Frameworks on CPAN • Maybe the best way to reuse them is to nick their ideas? • DBD::MysqlHA ? • Combination of DBIx::HA and DBD::Multiplex • MySQL specific: • Customizeable server selection algorithm • Persistent connections
Editor's Notes
- Start off by asking who knows what HA is, and get a feel for how much experience they’ve got with HA.
- Did your clients notice? Did you have an SLA? Were you in bed at the time? Did you have a backup server? Did you have a backup of the data? How long did it take to get the backup server with the backup data back online?
- HA for reads, but not for writes.
- Due to the way MySQL replication works - conflicting writes (typically auto increment keys) Mention FS replication
- Include further links
- Confirm with andrew on # requests a day.
- Speak about minimizing costs: replication across farms. Query caching run on all slaves out of memory to remove io bottleneck
- S/w Loadbalancing
- Mention Tear down / setup of connections.
- HA dsn: dsn1: Multiplex to primary master + many slaves dsn2: Multiplex to secondary master + many slaves If all slaves go down, will all reads go to master?