IPW2008 - my.opera.com scalability
•
5 likes•3,175 views
Opera Software is mainly known for its web browser, but Opera also created and develops a growing set of web applications like its own community dedicated to the browser, my.opera.com, started in 2001. Since then, the community grew bigger and bigger, and had the "usual" scalability problems. The application has been rewritten 3 times and the current developers team is at a turning point now. How to make the application and systems scalable with the growing amount of traffic and users?
Report
Share
IPW2008 - my.opera.com scalability
- 1. my.opera.com scalability v1 Italian Perl Workshop ~ Pisa 2008 cosimo streppone <cosimo@cpan.org>
- 4. Scalability == Performance
- 5. Scalability != Performance
- 6. Scalability == memcached?
- 7. Scalability != memcached
- 8. My Opera Community http://my.opera.com blogs shoutbox Opera browser promotion photo albums RSS-Atom feeds slideshows custom designs Public APIs xml/json forums
- 9. 2001
- 10. 2003
- 11. 2005
- 12. 2006
- 14. 2008
- 15. my.opera.com small (un)known web sites
- 16. Users (k) 2.500 * 1.640 IPW2008 887 430 257 205 1 10 50 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009
- 17. Users (k) Servers 2.500 * IPW2008 1.640 887 430 257 205 1 10 50 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009
- 18. Users (k) Servers Dyn req/s 2.500* 1.640 887 257 205 430 1 10 50 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009
- 19. proxy errors
- 20. the zen way to indefinite scaling
- 21. Start Y Does it run? Does it N N scale? No problem... Identify and fix Wait and restart Y bottlenecks End
- 22. /* recommendations */ server-side statistics good stress testing tools
- 24. shared-nothing architecture (“zero tolerance for sharing”)
- 25. lvs + lw httpd mod_perl ..... backends nfs server disk cache users store
- 26. lvs + lw httpd mod_perl ..... backends nfs server disk cache users store
- 27. lvs + lw httpd mod_perl ..... backends nfs server disk cache users store
- 28. ... not so easy ...
- 29. dev.opera.com articles cache (or “don't guess, profile!”)
- 30. /* problem */ 1 function (XWA::Quote2Dev::Quote) ~95% req time
- 31. /* solution */ Start Y Great. Serve In Cache? from cache. N Query DB, etc... Done! Put it in cache this time!
- 32. /* solution */ Yes, that simple! Profiling is key. try: m{^ Devel:: .* Prof .*}x
- 34. static avatars (or “put your http servers at work”)
- 36. /* problem */ a single forum page ~~ 50 avatars (crappy CGI backend requests)
- 38. new storage subsystem pools, servers fault tolerance, redundancy webdav, http, ftp, scp, mogilefs?, ...
- 39. user uploads use case # Create resource object for avatar my $res = MyOpera::Storage::Resource::Avatar->new( owner => '{userid}', content => '{binary data}', ); # Main storage subsystem handle my $storage = MyOpera::Storage->new(); # Upload on pools of servers all at once my $ok = $storage->upload($res);
- 40. resources (user uploads, binary blobs, ...) pools, servers URLs http://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_o.png http://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_t.jpg http://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_m.jpg http://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_l.jpg
- 41. /* results */ saved ~500k backend req/day browser cache used!
- 42. dogpile effect (or the cache “storms”)
- 43. /* problem */ very high load spikes on DB, backends
- 44. /* cause */ many concurrent clients build same cached items
- 45. /* solution */ cache contention algorithm NFS-resistant optimistic file locking
- 46. /* solution */ ideas taken from HTML::Mason, Django, ... patched File::NFSLock
- 47. /* further ideas */ LOCK_SPIN_TIME distributed cache on backends
- 48. Opera releases
- 49. /* problem */ 10-30x normal load handling of peak load?
- 50. /* try 1 */ load-balanced database profiles
- 51. /* try 2 */ url hot-list
- 52. /* solution? */ no solution this time, sorry.
- 53. /* further ideas */ Amazon EC2 ? Akamai CDN services? ... ?
- 54. soft counters (didn't work as expected, in progress...) sacrifice speed for concurrency (mysql's myisam locks all table on update)
- 55. /* future directions */ database partitioning? modperl backend optimization? threading sucks in mp2. debian anyone? improve memory sharing? sysadmin-level optimizations?
- 57. take-aways always compare before and after
- 58. take-aways stress testing in your release cycle
- 59. take-aways have fun! ;-)
- 60. ?questions?
- 62. thanks!