NPW2009 - my.opera.com scalability v2.0
- 4. Users (k)
2.500
1.640
887
430
257 205
50
10
1
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009
- 5. Users (k)
Servers
Dyn req/s
2.500
1.640
887
257 205 430
50
10
1
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009
- 7. espenao> proxys all over. MOC is dying?
hkmoen> did MOC just die?
...
fred joined #myopera
fred> lots of proxys...
fred left #myopera
...
cosimo> WTF? ok, looking into it
- 10. new storage subsystem
pools, servers
fault tolerance, redundancy
webdav, http, ftp,
scp, mogilefs?, ...
- 11. user uploads use case
# Create resource object for avatar
my $res = MyOpera::Storage::Resource::Avatar->new(
owner => '{userid}',
content => '{binary data}',
);
# Main storage subsystem handle
my $storage = MyOpera::Storage->new();
# Upload on pools of servers all at once
my $ok = $storage->upload($res);
- 12. resources
(user uploads, binary blobs, ...)
pools, servers
URLs
http://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_o.png
http://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_t.jpg
http://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_m.jpg
http://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_l.jpg
- 14. new cool static URLs
http://static.myopera.com/
/avatars/pool1/
/dL/rLA/vdUxYn14kfe342QEM/
/<user-id>_s.jpg
crappy CGI-style URLs
http://my.opera.com/<user>/avatar.pl
- 17. lvs +
lw httpd
mod_perl
.....
backends
nfs server
disk cache
users store
- 20. soft counters
use MyOpera::Counter;
my $counter = MyOpera::Counter->new({
key => “blog.access-$entry”,
sync => sub {
my ($value) = @_;
# Real table update code
...
}
});
$counter->inc();
- 21. dml statistics
[DML] time=1237308152, user=,
url=/tinh_yeu_cua_anh_b88/blog/index.dml/tag/...,
name=XWA::User, variable=active, type=module,
elapsed=0.068473, host=my.opera.com
[DML] time=1237308152, user=, url=/community/,
name=XWA::User, variable=, type=module, elapsed=0.015935,
host=my.opera.com
[DML] ...
- 22. top time-intensive modules
XWA::XMLProd::User::Sidebar 2024.919s (27.2%, 0.28 s/call)
XWA::User 1778.445s (23.9%, 0.09 s/call)
XWA::User::Journal 1121.224s (15.1%, 0.24 s/call)
XWA::User::Album 321.522s ( 4.3%, 0.17 s/call)
XWA::User::Journal::Search 223.477s ( 3.0%, 20.32 s/call)
XWA::User::Comments 188.011s ( 2.5%, 0.05 s/call)
XWA::Skins 180.486s ( 2.4%, 0.49 s/call)
XWA::User::JournalArchive 159.525s ( 2.1%, 4.43 s/call)
XWA::User::Posts 146.644s ( 2.0%, 0.45 s/call)
XWA::User::Picture 141.324s ( 1.9%, 0.10 s/call)
XWA::Albums 93.740s ( 1.3%, 2.04 s/call)
XWA::Journals 92.390s ( 1.2%, 2.37 s/call)
- 23. most called modules
XWA::User 18784 calls 28.9%
XWA::XMLProd::User::Sidebar 7186 calls 11.1%
XWA::User::Spotlight 6439 calls 9.9%
XWA::Util::OneTimeKey 5576 calls 8.6%
XWA::User::Journal 4681 calls 7.2%
XWA::OIVWrapper 3974 calls 6.1%
XWA::User::Comments 3890 calls 6.0%
XWA::User::Album 1881 calls 2.9%
XWA::User::Ratings 1500 calls 2.3%
XWA::User::Picture 1485 calls 2.3%
XWA::XMLProd::MemberCount 611 calls 0.9%
XWA::XMLProd::Queue 589 calls 0.9%
XWA::Spotlight 569 calls 0.9%
MyOpera::DML::FastFacts 567 calls 0.9%
XWA::XMLProd::Variables 562 calls 0.9%
- 24. slowest modules
XWA::User::Journal::Search 20.32s (11 calls)
XWA::User::JournalArchive 4.43s (36 calls)
XWA::User::GeneratePassword 3.76s (3 calls)
XWA::XMLProd::Link::Bookmarks 3.45s (12 calls)
XWA::Fans 3.37s (10 calls)
XWA::User::ForumLatestFriends 3.36s (11 calls)
XWA::User::ForumNew 2.42s (3 calls)
XWA::Journals 2.37s (39 calls)
XWA::Albums 2.04s (46 calls)
XWA::XMLProd::UserSearch 1.87s (2 calls)
XWA::Jahoo 1.83s (46 calls)
XWA::Forum 1.64s (49 calls)
XWA::XMLProd::Link::Trash 0.86s (2 calls)
XWA::XMLProd::LocationSearch 0.72s (43 calls)
XWA::User::ForumSearch 0.67s (78 calls)
- 26. low level TCP/IP “throttling”
# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all -- x.x.x.x/x anywhere
limit: avg 20/sec burst 5
ACCEPT all -- y.y.y.y/x anywhere
limit: avg 20/sec burst 5
DROP all -- x.x.x.x/x anywhere
DROP all -- y.y.y.y/y anywhere