Boomerang @ NY Web Perf meetup

Why measure?
boomerang
data data data

Measuring the web with boomerang

Philip Tellis / philip@bluesmoon.info

NY Performance Meetup / 2010-09-15

NY Performance Meetup / 2010-09-15 Measuring the web with boomerang

Why measure?
boomerang
data data data

$ ﬁnger philip

Philip Tellis
philip@bluesmoon.info
@bluesmoon
yahoo
geek
http://bluesmoon.info/


Why measure? The slow web
boomerang Measurements
data data data Measuring with javascript

Where does all the time go?



Who controls it?

Some of this we control and some of it we don’t



Back end

Measuring and improving back end performance can be done
during development



80-20

Turns out that less than 20% of the time is spent on the back
end



Front end

It’s what we can’t control that bites us



Too many variations

browsers



Too many variations

plugins



Too many variations

OSes



Too many variations

viruses



Too many variations

antiviruses



Too many variations

microwaves



Too many variations

baby monitors



Too many variations

naughty neighbours



Too many variations

ﬁle shares



Too many variations

governments



Too many variations

rodents



Too many variations

Try simulating all that in the lab!



We need to measure real end-user performance



We need to measure it from the real end-user’s box



Ask the user?



Bias

While this might work, it isn’t necessarily representative



A/B testing

You also want to be able to dynamically tune which users get
which tests



Phone home

It’s most useful if you can send these measurements back to
your server for analysis



Mostly ubiquitous

We know that javascript is available on almost every browser



Rich pages

We really want to measure the performance of rich pages
which depend on javascript already



Limited

But javascript can’t measure everything... we get as close as
we can


Why measure? What?
boomerang How does it work?
data data data Accuracy

A piece of javascript that you add to your web page where it
measures and beacons back to you, the end user’s perceived
performance of your page


Why measure? What?

How?

<script src="boomerang.js" type="text/javascript">
</script>
<script type="text/javascript">
BOOMR.init({
user_ip: "<user’s ip address>",
beacon_url: "http://yoursite.com/beacon.php"
});
</script>


Why measure? What?

What does it do?

About once a week, measures user’s bandwidth and
latency to your server
On (almost) every request, measures the time it took to
load the current page
Beacons these results back to your server
Other stuff based on plugins


Why measure? What?

How does it do it?

Let’s take that one at a time


Why measure? What?

How do we measure latency?

Download a 32 byte gif 10 times in sequence
Measure the time to download each
Discard the ﬁrst measurement because it’s overpriced
Calculate the arithmetic mean, standard deviation and
margin of error of the rest


Why measure? What?

Wait, did you say overpriced?

The ﬁrst image might require a DNS lookup and TCP
handshake
Slow start is not an issue since 32 bytes ﬁts in 1 packet


Why measure? What?

How do we measure bandwidth?

After the latency test is done, we download progressively
larger images
Stop at the ﬁrst image that times out
Redownload that image a few more times
Calculate the median, standard deviation and margin of
error of the largest images


Why measure? What?

Measuring latency before bandwidth helps here

Those 10 latency images do a lot to widen the TCP
window size
The bandwidth images make much better use of bandwidth
The image we end with uses the most bandwidth


Why measure? What?

How do we measure page load time?

In the onbeforeunload event, measure the time and
store it in a cookie
In the onload event, check the cookie, and measure the
difference with the current time
We also make sure that the page that set the cookie is the
referrer of the current page


Why measure? What?

What? Two pages?

Yes, this needs two pages and cookies. If those aren’t
supported, we try to use the WebTiming API.


Why measure? What?

How accurate is it?

Latency measurements are very accurate (±1%)
Bandwidth is to an order of magnitude. For bad
connections can be ±30%
Page load time sometimes has outliers, you need
post-ﬁltering
The margin of error tells you how good your data is


Why measure? Filtering
boomerang Grouping
data data data Data

What do we do with the data?

Sanity checking to:
Remove fake data
Remove abusive data
Maybe just rate limiting


boomerang Grouping
data data data Data

What do we do with the data?

Statistical analysis to:
Remove outliers
Aggregate based on bandwidth blocks
Measure trends over time and correlate them with code
changes


boomerang Grouping
data data data Data

Bandwidth blocks

0-100 kbps
100-300 kbps
300-2000 kbps
2-6 Mbps
6+ Mbps


boomerang Grouping
data data data Data

Bandwidth blocks

Group page load times based on bandwidth block


boomerang Grouping
data data data Data

Bandwidth blocks

Data points from some countries may require narrower bands


boomerang Grouping
data data data Data

Geographic data

Looking at latency from different geographic locations can tell
you where to put your next CDN


boomerang Grouping
data data data Data

ISPs

Grouping data by ISP can tell you who’s behaving badly


boomerang Grouping
data data data Data

Storing the data

We log all beacon requests to apache’s log access ﬁle
Low trafﬁc sites could write directly to a DB
Others have suggested using CouchDB as the beacon
server
Daily summaries can be sent across to ShowSlow


boomerang Grouping
data data data Data

More data

Write plugins to get more performance data
We already have a DNS plugin
I’m thinking of an IPv6 v/s IPv4 plugin
What about a full WebTiming plugin?
Can we measure connection setup time?


boomerang Grouping
data data data Data

You decide

Once you have the data, you can do anything with it


boomerang Grouping
data data data Data

Thank you
http://github.com/yahoo/boomerang
http://yahoo.github.com/boomerang/doc/


boomerang Grouping
data data data Data

Photo credits

ﬂickr.com/photos/21233184@N02/4389412851


boomerang Grouping
data data data Data

Contact me

Philip Tellis
yahoo
geek
@bluesmoon
http://bluesmoon.info/
slideshare.net/bluesmoon
philip@bluesmoon.info


boomerang Grouping
data data data Data

References

github.com/yahoo/boomerang
More bandwidth doesn’t matter (much) – Mike Belshe
Analysing Bandwidth & Latency – YUI Blog
It’s the latency, stupid – Stuart Cheshire
The statistics of web performance


Boomerang @ NY Web Perf meetup

More Related Content

More from Philip Tellis

More from Philip Tellis (20)

Recently uploaded

Recently uploaded (20)

Boomerang @ NY Web Perf meetup