Scalability at GROU.PS
- 2. DisclaimerWe’re not fully there yetWe hire: jobs@groups-inc.com
- 3. Challenges @ GROU.PS3M unique visitors per month120M page views1PB assets to be served every monthVideo,Photos, FilesSupport for 5Gbit/sVery dynamic pages:With social networks; p(u,t) = HTMLp(g,u,t) = HTML -> WHERE group_id = ? AND …
- 4. What is GROU.PS ?
- 9. Distributed Architecture25+ servers, S3 cloud, EdgeCast CDN4 cores + All Linux: Red HatSome Debian, Ubuntu, CentOS
- 10. Amazon TechnologiesS3CloudFrontEC2 (elastic IP and persistent storage)SimpleDBQueue technologies, distributed hadoop and more…
- 11. Amazon TechnologiesDownside: Not so cheapBad database performance
- 12. Serving Content?Use MogileFSDistributed file servingUse CDNhot content served off from local serversSysctl tunings needed!
- 13. Our typical sysctl additionsnet.ipv4.tcp_syncookies = 1net.ipv4.tcp_synack_retries = 2## Emre edited# http://www.oracle-base.com/articles/11g/OracleDB11gR1InstallationOnFedora8.phpkernel.shmall = 2097152kernel.shmmax = 2147483648kernel.shmmni = 4096# semaphores: semmsl, semmns, semopm, semmnikernel.sem = 250 32000 100 128net.ipv4.ip_local_port_range = 1024 65000net.core.rmem_default=4194304#net.core.rmem_max=4194304net.core.wmem_default=262144#net.core.wmem_max=262144fs.file-max=5049800vm.swappiness=10## Emre edited# from http://forums.softlayer.com/showthread.php?t=3252net.ipv4.tcp_rmem = 4096 87380 8388608net.ipv4.tcp_wmem = 4096 87380 8388608net.core.rmem_max = 8388608net.core.wmem_max = 8388608net.core.netdev_max_backlog = 5000net.ipv4.tcp_window_scaling = 1net.ipv4.ip_nonlocal_bind=1# http://rackerhacker.com/2007/08/24/apache-no-space-left-on-device-couldnt-create-accept-lock/kernel.msgmni = 1024kernel.sem = 250 256000 32 1024net.ipv4.ip_conntrack_max = 524288net.ipv4.netfilter.ip_conntrack_max = 524288
- 14. MySQLLoad off via memcache$memcache->set(“group_by_name.jtpd”, 1122, false, 0);$memcache->set(“home_module_html.1122”,…, true, 30);function getGroupID($group_name) { global $memcache; if( !isset($memcache) || ($res=($memcache->get(“group_by_name.{$group_name}”)))===false ) { // get it from mysql and memcache } else { return $res; // serve from memcache }}
- 15. MySQLReplication easySplit ReadsWhat about writes?That’s where sharding comes to playVertical ShardingHorizontal ShardingMMM
- 16. MySQLRuns poorly on multi-coresquery_cache_size = 0 # on masterquery_cache_type = 0 # on masterthread_concurrency = 8 # total coresmax_connections = 750 # shouldn’t exceed thatinnodb_buffer_pool_size = 10G # a little less than the total amount
- 17. MySQL Query OptimizationINDEX group, userWHERE group = ? AND user = ?Not WHERE user = ? AND group = ?B-tree
- 18. MySQL Query OptimizationSHOW PROCESSLISTMaatkit, mk-query-digestPercona builds
- 19. NOSQLVoldemort, LinkedinCassandra, FacebookTokyo Cabinet, mixi
- 20. LoggingDatabase logging is not the solutionFile system is expensive tooA legal necessity
- 21. LoggingSolution:Scribe & ThriftBy FacebookEventually consistent
- 22. Nginx & libevent
- 23. Nginx & libeventHandles 10000 connections5gbit/sRamblerWordpressGrou.ps
- 24. PostfixRun multiple instancesSpam Clusters
- 25. MonitoringMunin + monitOther alternatives:CactiNagiosHyperic – vmware
- 26. PHP
- 27. More to come on my bloghttp://emresokullu.comMore fine tuning tipsBecome a member of my communityLove grou.ps ;)Convert to PHPWe’re hiring: jobs@groups-inc.com