DIY Synthetic: Private WebPagetest Magic

DIY Synthetic:
Private WebPagetest Magic
Velocity NY 2013
Jonathan Klein
@jonathanklein
Wednesday, October 16, 13

Slides, Links
jkle.in/wpt


About Me


About Me
• Performance Engineer At Etsy


About Me
• I write the Etsy Site Performance Reports


About Me
• I write the Etsy Site Performance Reports
• Organize Boston Web Perf Meetup Group


Etsy Stats


Etsy Stats
• World’s largest handmade marketplace


Etsy Stats
• 1.5 billion page views/month


Etsy Stats
• 1.5 billion page views/month
• Almost $1B in sales last year


+160k to a page =
+12% bounce rate
on mobile

Synthetic Monitoring


Synthetic Monitoring
Browser Instrumentation

Navigation Timing API

Consistent trending
over time

Highly variable

Largely in your control

Last mile difﬁculties

Great for identifying
regressions

Great for comparing across
geographies/browsers

Not super realistic


Real User Monitoring

“Real User Monitoring”

$$$$


Background


EC2 Test Agents


No Built-in Automation


wpt-script


Motivation


Why We Use It


Why We Use It
• Wanted long term synthetic trending


Why We Use It
• Automated tests


Why We Use It
• Automated tests
• More control


Why We Use It
• Automated tests
• More control
• Low cost


Easy Configuration


{
    "server": "http://www.yourwptinstall.com",
    "pending_dir": "private",
    "locations": ["US_East_wptdriver:Firefox", "US_East_wptdriver:Chrome"],
    "urls": {
        "label": "http://www.example.com",
        "other_label": "http://www.anotherexample.com"
    },
    "graphite": "graphite.example.com",
    "logging_ns": "webpagetest.private",
    "splunkLog": "/var/log/webpagetest/results.log"
}


# Run Test
*/25 * * * * php run.php


# Run Test
*/25 * * * * php run.php -c foo.conf


# Run Test
*/25 * * * * php run.php -c foo.conf
# Get Results
* * * * * php get_results.php


Advanced Features


{
    "server": "http://example.com",
    "urls": {
        "other_label": "http://www.webpagetest.org"
    },
    "splunkLog": "/var/log/webpagetest/results.log",
    "prepend": "Login",
    "username": "someuser",
    "password": "somepassword",
    "run_options": {
        "video": 0
    }
}


{
    "server": "http://example.com",
    "urls": {
        "other_label": "http://www.webpagetest.org"
    },
    "splunkLog": "/var/log/webpagetest/results.log",
"prepend": ["BlockThirdParty", "bypass_cdn.txt"],
    "run_options": {
        "video": 0
    }
}


foreach ($prepends as $prepend) {
if (file_exists($scripts_dir . $prepend)) {
$script .= file_get_contents($scripts_dir . $prepend);
} elseif (method_exists($this, $prepend)) {
$script .= $this->$prepend();
}
}


bypass_cdn.txt
setDns
setDns
setDns
setDns
setDns
setDns
setDns
setDns
setDns


site.etsystatic.com 38.106.64.123
img0.etsystatic.com 38.106.64.125
ny-image0.etsy.com 38.106.64.125

Gathering Results


<?php
// Splunk
$splunkLogger = new SplunkLogger($config['splunkLog'], $logging_ns);
foreach ($results as $result) {
$splunkLogger->log($result);
}
// Graphite
$grapher = new Grapher($config['graphite'], $logging_ns);
$grapher->graphResults($results);


https://sites.google.com/a/webpagetest.org/docs/using-webpagetest/metrics

Disadvantages vs. Paid Tools


• Harder to set up/maintain


• Functionality


• Functionality
• Regions


• Functionality
• Regions
• Full page oriented


• Functionality
• Regions
• Full page oriented
• Less consistent


Advantages vs. Paid Tools


• Cost


• Cost
• Flexibility


• Cost
• Flexibility
• People know the tool


• Cost
• Flexibility
• Multi-page flows


• Cost
• Flexibility
• Scripting


Our Use Cases


WebPagetest


Catchpoint

WebPagetest
• Scripted tests


Catchpoint

WebPagetest
• Scripted tests
• High volume testing


Catchpoint

WebPagetest
• Scripted tests
• Ad-hoc tests


Catchpoint

WebPagetest

Catchpoint


• DNS

• Scripted tests
• Ad-hoc tests


WebPagetest

Catchpoint


• DNS

• Scripted tests

• API

• Ad-hoc tests


WebPagetest

Catchpoint


• DNS

• Scripted tests

• API


• Single Object Tests

• Ad-hoc tests


WebPagetest

Catchpoint


• DNS

• Scripted tests

• API



• Ad-hoc tests

• Multiple locations


WebPagetest

Catchpoint


• DNS

• Scripted tests

• API



• Ad-hoc tests

• Multiple locations
• Comparisons


Future Work


Improving the Process


• Manage disk space


• Multiple regions


• More canned scripts


• More canned scripts
• Automate upgrades


Leveraging WebPagetest


• Front-end SPOF


• Front-end SPOF
• User Timings (AFT)


• Front-end SPOF
• User Timings (AFT)
• Speed Index


Improving
WebPagetest


WebPagetest + Vagrant


• Install VirtualBox


• Install Vagrant


• Install Vagrant
• https://gist.github.com/jklein/6992296


• Install Vagrant
•vagrant up


• Install Vagrant
•vagrant up
• http://localhost:8080


800MB


webpagetest-api


Both Are Good Options


Results > Tools


Thanks!


Get in Touch
@jonathanklein
jonathan@etsy.com
www.etsy.com/careers

DIY Synthetic: Private WebPagetest Magic

More Related Content

DIY Synthetic: Private WebPagetest Magic