Микола Ковш “Performance Testing Implementation From Scratch. Why? When and How?”

1
Skills | Knowledge | CollaborationSkills | Knowledge | Collaboration
PerformanceTestingImplementationFromScratch.Why?
WhenandHow?
by Mykola Kovsh

2
Skills | Knowledge | Collaboration
Speaker info
Hi, I’m Mykola Kovsh
PerformanceTestEngineer
PerformanceTesting Departmentat
Testing Centerof Excellence,
Ciklum, Ukraine
•4 yearsin IT
•Performance/ Automation/ Manualtesting
www.linkedin.com/in/mykolakovsh/

3
Content
Performance testing engineer skills5
Why performance testing is important?1
Introduction to performance testing. Load profiles, metrics2
Implementation process4
Performance testing tools3
Links and Q&A session

4
1. Why performance testing is important?

5
Why performance testing is important
✓ 50% of frustrated users will visit another
website to accomplish their activity and 22%
won't return.
✓ 49% of respondents expect web pages to
load in under 2 seconds.
✓ 30% expect a 1-second response.
✓ 18% expect a site to load immediately.*
* http://docplayer.net/29696161-Performance-matters-key-consumer-insights.html

6
From user perception point of view
What is a fast web-side?
< 1 second very fast
< 2 seconds quite fast
2-4 seconds acceptable
5-15 seconds slow
>15 seconds too slow
* https://www.hobo-web.co.uk/your-website-design-should-load-in-4-seconds/

7

8
How smartphone users react to slow web-sites
Curse at their
phone, 23%
Scream at
their phone,
11%
Throw their
phone, 4%
Behave more
or less
normally,
62%

9
Cost of poor performance
* http://www.webperformancetoday.com/2010/06/15/everything-you-wanted-to-know-about-web-performance/
 If your average sales per hour is
$5,000
1 minute of downtime is
costing over $80
then

10
 A 1-second page load delay equals*:
16 % decrease in customer
satisfaction
11 % fewer page
views
7 % loss in
conversions

11
For example:
✓ John Lewis’s website went down around 3.20 pm on Black Friday - analysts estimated it could
cost £ 2.8m pounds.
✓ Web giant like Amazon would lose as much as $120 000 per minute of downtime.
✓ Microsoft Bing found that a two-second slowdown caused a 4.3 percent reduction in
revenue per user.
✓ Website Shopzilla reduced page load times from 7 seconds to 2 seconds, resulting in a 7–12
percent increase in revenue and 50 percent reduction in hardware costs.

12
When performance testing is needed
✓ The solution is already struggling from performance
problems.
✓ The solution have to deal with big amount of concurrent
users presently or in future (~ 300-500 concurrent users and
higher).
✓ The solution have a large database or should transfer or
process big amount of data in real time (~ 10 and more
concurrent users).
✓ The solution has complex architecture and a lot of internal
and external integrations running concurrently.

13
2. Introduction to Performance Testing

14
✓ Performance Testing - the process of testing to determine the
performance of a software product (ISTQB Foundation).
✓ Performance testing – is a non-functional software testing technique
which determines responsiveness, stability, reliability and resource
usage of system under a certain user load (Wikipedia).
✓ Web load testing is nothing more than exercising a website under a
variety of production-like conditions to determine how it’s going to work
and to identify (and hopefully resolve) problems before your customers
find them (WebLoadTestingForDummies).
Performance testing. Definition

15
Determine responsiveness, stability, reliability and resource usage of system
under a certain user load
Demonstrate that the system meets performance criteria (KPIs)
Compare different system configurations and versions to evaluate
performance improvement/degradation.
Performance trends tracking during the time
Determine system behavior under different load
Evaluate the system capacity
Scalability. Determine ability of a system to handle a growing amount of workload
Prepare the application for planned load (e-commerce: Black Friday, Marketing
Campaign. Finance: quarter/annual reporting, etc.)
Find which components of the system perform poorly under certain workload
Goals of performance testing:

16
Slow sub-systems / functions (poor response)
Low capacity point
Configuration problems (web-server,
load balancers, db etc)
Dead-lock while simultaneous load
Flawed queue logic
Incorrect synchronization of recourses
Database issues e.g. size, indexing, replication
Memory, space and connections leaks
Poor network configuration
CPU, Memory utilization
Functionality bugs (how system should behave under overload, others).
Performance bottlenecks:

17
Functional & Performance testing comparison
# Functional testing Performance testing
1
To verify the accuracy of the system against
expected results
To verify the behavior of the system at various
load conditions
2 Manual or automated Automated only
3 Could be done without special tools
Special set of tools is used including analyzing
and monitoring ones
4 One user performing all operations Several users performing desired operations
5
Involvement required from Customer, Tester
and Developer
Involvement required from Customers, Tester,
Developer, DB admins, DevOps
6
Test environment capacity/size could differ
from Production
Requires close to Production Test
environment!!!
based on: http://www.softwaretestinghelp.com/introduction-to-performance-testing-loadrunner-training-tutorial-part-1/

18
Performance testing Approaches
Protocol-level Load Generation
(HTTP, Web-sockets, TCP, JDBC,
etc.)
Device level Load Generation
(User Interface)

19
Performance testing Approaches
Protocol-level Load Generation
(HTTP, Web-sockets, TCP, JDBC,
etc.)
Device level Load Generation
(User Interface)

20
Performance testing Targets
Server-side performance (web, app, database)
Client-side (manual) performance
Network performance
Device performance (hardware: battery consumption, CPU, memory, etc.)

21
Performance testing Targets
Server-side performance (web, app, database)
Client-side (manual) performance
Network performance
Device performance (hardware: battery consumption, CPU, memory, etc.)

22
Server-side performance

23
Levels of server-side performance testing
• End-to-End load testing
• Integration Load testing
• Component/API Load testing
• DB performance testing

24
Types of Performance Testing (load profiles)
Stress/Capacity test
Max Designed Operation Capacity
Volume test
8-72 hours or longer
+ Component Test
+ Reliability /
Recovery Test
Server-side performance

25
Client-side (manual) Performance

26
Client-side performance testing - testing of one separate page load from client/browser side
Client-side Performance

27
More detailed process of page load

28
First thing the user sees

29
Visual Experience
1. First Paint 2. First Contentful Paint
3. First Meaningful Paint 4. Visually Complete
based on: https://www.slideshare.net/nicjansma/measuring-real-user-performance-in-the-browser

30
Chrome performance testing

31
1. Run client-side performance of the page when there is 0 load.
2. Set performance metrics baselines based on step 1
3. Run client-side performance when there is different load on the server –
during Capacity/Load/Spike etc. tests
4. Compare results with baselines, analyze and summarize possible issues
Flow we suggest:

32
Load testing metrics

33
Load testing metrics (synthetic monitoring)
Application-side Server-side
- Response time
- Throughput (rps/tps/tpm)
- Concurrent users
- Error rate (response code)
- Number of transactions passed/failed
- Network traffic
- CPU
- Memory
- Network
- Disk
- DB connections
- Logs error, warnings

34
Response time = Latency (travelling across a network) + Processing time (system processing
of request)
• Average response time
• Peak response time (max)
• Response time with 95% or other percentile
Response time
Application-side metrics

35
Throughput - how many simultaneous
requests/transactions per second/minute application can
handle
Note:
 TPS could correlate with response time if requests are
consequent. The longer response – the lower tps
 TPS does not correlate directly with response time if
requests are parallel.
consequent > 1 request – 1 sec response – 1 tps
parallel > 10 requests – 1 sec response – 10 tps
 TPS could be improved by improving response time or
by increasing concurrent users
Throughput (rps/tps/tpm)

36
Network traffic, Response codes, Error rate
Network traffic – shows how much data is flowing back
and forth from your servers (Kbytes or Mbytes / sec).
We can compare this metric to the response-time metric
to see how the throughput affects transaction
performance.
Error rate – is the mathematical calculation that
produces a percentage of problem requests compared to
all requests.
It is no standard for tolerable error rate. Some projects
consider 1% error rate successful in case the system can
handle maximum load without crash. Others consider
any errors. Still, few errors is not uncommon, especially
for large load.

37
• You can monitor server-side metrics directly on the server (Linux, Windows)
• You can automate this process creating some monitoring agent to track metrics
• You can use one of monitoring tools:
- CloudWatch (Amazon)
- AppDynamics
- DynaTrace
- NewRelic
- Graylog, etc.
Server-side monitoring tools
*In addition to the server metrics monitoring during the load test (synthetic monitoring), monitoring tools allow
Real User Monitoring (RUM).
RUM is a type of performance monitoring that captures and analyzes each transaction by real users of a website or
application. Unlike synthetic monitoring, RUM never rests. It collects data from each user using every browser
across each request.

38
3. Performance testing tools

39
Strong load testing tools should be able
Traffic recorder
Have IDE (console or GUI) which allows:
a) Create Requests of required protocol (HTTP, HTTPs, WS, WSS, JDBC, TCP, AJAX, etc.)
b) Support Transactions – to track time for all static data loading / redirections
c) Create Load Scenarios with ability of parametrization
d) Build different Load Profiles with rump-up and shut-down
e) Have debugger
Load runner engine
Distributed testing
Load test data saving (distributed), including client and server-side metrics
Load test data monitoring in real time

40
- JMeter
- Gatling
- Locust
- The Grinder
- Apachebench
- Artillery
- Tsung
- Vegeta
- Siege
- Boom
- Wrk
Open source load testing tools
According to Load Impact tools comparison research*:
 JMeter, Gatling, Grinder, Tsung and Boom all offer good performance, accuracy and reliability
 Artillery, Locust and Siege have various issues with performance, accuracy and/or reliability
 Performance-wise, Wrk and Apachebench are in a class of their own
 NOTE: None of the tools tested can simulate thousands of VUs on a single machine without significant
degradation in measurement accuracy
* http://blog.loadimpact.com/open-source-load-testing-tool-benchmarks

41
Cloud-based tools:
- BlazeMeter
- LoadImpact
- Loadstorm
- Loader.io
- Loadfocus, etc
Commercial load tools
Commercial Combiners:
- HP Loadrunner
- Microsoft TFS
- SmartBear LoadUI
- NeoLoad
- Silk Peformer

42
• Chrome developers tools
• http://yslow.org/
• https://tools.pingdom.com/
• https://www.webpagetest.org/
• https://developers.google.com/speed/pag
espeed/insights/
• https://www.sitespeed.io/ (automatically
run using Docker)
• Performance monitoring and management
systems also can track timing metrics of
the page load (RUM), like NewRelic
Manual performance testing tools
YSlow test example:

43
4. Implementation process

44
Performance testing process activities
1. Planning
2. Implementing
3. Executing
4. Analyzing and
reporting
5. Continuous
Integration

45
1. Planning
Identify Performance Acceptance Criteria and KPIs
• Response time for different type of transactions -> a user concern
• Throughput (tps/rps) -> a business concern
• Resource utilization -> a system concern
• Concurrent users number
• Accepted Error rate
• Accepted deviation for response time, resource utilization
• System behavior when overloaded

46
1. Planning
Performance requirement analysis
Bad sample of requirements:
1. Response time should be no more than 4 seconds
2. System should be able to deal with 10 thousands
concurrent users
It is not clear whether:
• Response time should be 4 seconds for all requests?
• What if all response time will be 3.99 seconds?
• What if most request will response with 2 seconds but several with 6 seconds?
• What users do and how often? Open Main page or more?
• What if more than 10 ths. users? System should scale, or new users will be rejected or response time
just increases?

47
1. Planning
Performance requirement analysis
Good sample of requirements:
1. Set average response time for different types of transactions:
• Simple navigation requests - 1 second
• Logging - 2 seconds
• Search and buy - 4 seconds
2. Set deviation - no more than 15%
3. Set failure rate - no more than 1%
4. Set CPU, memory, network, other server-metrics thresholds
5. Set response times, deviation, server-side metrics for different number of users - for
1k, 5k, 10k
6. System should reject new users in case of overload, showing informing message

48
1. Planning
Identify the Test Environment
• Identify the logical and physical production architecture for performance testing
• Compare the both test and production environments while identifying the testing environment
TEST environment MUST BE the same as PROD (or run tests on PROD)
• Get resolve the environment-related concerns if any – using stabs for 3-rd parties or others
• Analyze whether additional tools are required for performance testing, like monitoring tools. Install such
tools.
Identify scope of load testing (product parts, 3-rd party services in/out of scope)
Identify technical nuances
• Scheduling services?
• Ping calls
• Client’s internet connection speed?
• Static content hosting: CDN or own servers?
• Target region (USA, Europe, etc.)?

49
1. Planning
Plan and Design Tests
• Identify key usage scenario and workload (Load, Capacity, Spike, etc.)
• Define test data
• Establish metrics to be collected
What transactions to include:
1. Critical transactions
Example: System login, session support
2. Mostly used transactions by real users
Example: System login/logout, main page
3. Business required
Example: some specific feature
4. Risky transactions
Example: Checkout, payment
5. Heavy transactions
Example: File download/upload

50
User Community Modeling Language (UCML) for Performance Test Workloads
1. Planning

51
Prerequisites solving
1. Planning
• Add your load generator machine IP(s) into whitelisting (if needed)
• Request test users or create them by yourself
• Request test data (products names, files, etc.)
• Request 3-rd party dependent data (payment cards, etc.)
Configure load-generation environment
The output of this stage is prepared Performance Test Plan
• Environment capacity (CPU, memory) should be enough to run required number of users during
some time
• Setup Master-Slave architecture in case of distributed load testing
• Setup monitoring tools to track load server(s) health during tests running

52
2. Implementing
• Develop performance scripts
• Put assertions points and wait timers to make it a real time scenario
• Run several smoke runs to calibrate scripts to the target environments in accordance with test design
Simulate real users behavior basing on Production usage statistic

53
3. Executing
It very depends on a project, but still we can suggest next
How to choose and run correct load profiles

54
0. Warm up your product servers before load tests run
- Run some short smoke load test for several users
3. Executing

55
1. Run several baseline Load tests to determine benchmark performance metrics
levels:
- virtual users: 5-100
- duration: 30-60m
- rump-up – 10m; run – 20m; shutdown – 5m
!!! Run each test at least 2 but better 3 times to be sure in results
3. Executing

56
2. Run the Capacity test profile and determine when Capacity point (system limit)
happens:
- virtual users: a lot, depends on the system
- duration: 2-3 hours
- rump-up – 2-3 hour; run – NA; shutdown – NA
3. Executing

57
How to understand that you reach Capacity point:
When transactions Response time increases sharply with load increase.
Note:
- although load increase, response time should not increase sharply. However, small
deviation is acceptable;
- response time also could start decrease dramatically, what means that you start receive
responses with errors.
When the Error rate increases with load increase
Note: often, small error rate is acceptable (1-3%), especially during large load tests.
Crash of the servers or one of them (web, application, DB, etc).
3. Executing

58
Capacity point example
V users: 10 ths.
Status: failed
Error rate: test stopped on 28%

59
V users: 9 ths.
Status: failed
Error rate: test stopped on 49%

60
V users: 8 ths.
Status: passed
Error rate: 0.21% (500 errors
from 236 ths. transactions

61
3. Run the Load test profile with:
- virtual users: 50-80% of Capacity point or/and required load number
- rump-up – 60m; run – 60m; shutdown – 20m
3. Executing

62
4. Run Component tests for each service / micro-service / function separately:
- could be run before or after Endurance test, especially if issues for some service(s) were
found
- virtual users: firstly, like for Capacity test, then like for Load profile
- rump-up, run and shutdown timings depend on component test profile
3. Executing

63
5. Run the Endurance test profile:
- virtual users: 20-40% of Capacity point
- rump-up – 30m; run – 23р; shutdown – 30m
3. Executing

64
6. Run other types of load testing and play with your scenarios combinations.
- volume test, Spike, Reliability/Recovery, Volume tests
- each of these tests could be run on the first stages, if it is prioritized be business
- run load tests for different user flows combinations
3. Executing

65
• In ongoing development -> Verifying and validating
component, queue of components, and integration
related performance & robustness
• Before release -> Verifying and validating the whole
product performance & robustness before release.
• Maintenance -> Verifying and validating architectural,
configurational, capacity-related, db-related, and
integration-related changes
When to perform
3. Executing

66
• Collect and analyze load tests data
• Investigate possible bottlenecks (memory, disk, processor,
process, cache, network, etc.) resource usage like (memory,
CPU, network, etc.)
• Generate the Performance analysis reports
Note: The report form is
- a performance test summary report
- transactions details, hardware utilization, etc.
- the comparison of actual and expected KPIs
• Based on the analysis prepare recommendation report
• Share report with the team and stakeholders
4. Analyzing and reporting

67
Step 9 – Report example

68
5. Continuous Integration
or
Development
Continuous Integration
Build. Unit test
Deploy to Dev Server
Integration Test
Deploy to Test/QA
Server
Automated Functional
Tests
Deploy to Load Server
Performance test in CI +
monitoring
Manual QA Test
Promote to
Stage / Pre-Prod
Ideally, if you have separate
environment for load testing
Note: you can implement
load tests on Stage too, if
it is the same as Prod

69
5. Performance testing engineer skills

70
Performance testing engineer skills
Goodenough programmingskills (Python,Java,C++etc.)
Loadtoolsandtheirsadvantages/disadvantages(JMeter, LoadRunner,Gatling etc.)
RUM tools(NewRelic,AppDynamics,CloudWatch,Zabbixetc.)
CI integrationtools
Tools knowledge
Knowledgeof performancemeasurementsandmetrics.Abilitytocalculate/interpretthem
The client performancerequirements andgoals analysis
Identifyingtransactionsandworkflows– calculatingworkloadTPSgoals andrates
Performance theory knowledge

71
Experiencein differentOSadministration:Windows,Linuxdistributives
Interpretanddrawsystemsdiagrams
Basicunderstandingof systemsenvironments– sharedresources,components,andservices
Thedifferencesbetween productionandtestenvironments– containers,cloud, virtualization,andconfigurationmanagement
RelationalandNoSQLdatabases
ContinuousIntegrationandDeploymentconcepts
Systems and architecture knowledge, DevOps

72
Projectmanagementskills
Communicationskills
Eagerforconstantself-development andimprovement. Open fornew technologies
Soft skills

73
Useful Links
About performancetesting:
http://docplayer.net/29696161-Performance-matters-key-consumer-insights.html
http://www.softwaretestingclass.com/what-is-performance-testing/
https://www.blazemeter.com/blog
http://revolutionit.com.au/performance-testing-digital-projects-dont-leave-it-to-the-developers/
Concurrentuserscalculation:
https://blog.xceptance.com/2013/07/26/concurrent-users-the-art-of-calculation/
https://techbeacon.com/how-many-virtual-users-do-i-need-load-testing
https://www.webperformance.com/library/tutorials/CalculateNumberOfLoadtestUsers/
Browserperformance:
https://www.slideshare.net/nicjansma/measuring-real-user-performance-in-the-browser
Booksfor start:
“Web Load Testing ForDummies”,Scott BarberwithColin Mason
“JMeterCookbook”, Bayo Erinle

74

Микола Ковш “Performance Testing Implementation From Scratch. Why? When and How?”

More Related Content

What's hot

What's hot (20)

Similar to Микола Ковш “Performance Testing Implementation From Scratch. Why? When and How?”

Similar to Микола Ковш “Performance Testing Implementation From Scratch. Why? When and How?” (20)

More from Dakiry

More from Dakiry (20)

Recently uploaded

Recently uploaded (20)

Микола Ковш “Performance Testing Implementation From Scratch. Why? When and How?”

Editor's Notes