More Related Content
What's hot
What's hot (20)
Similar to PAC 2020 Santorin - Giovanni Paolo Gibilisco
Similar to PAC 2020 Santorin - Giovanni Paolo Gibilisco (20)
More from Neotys
More from Neotys (20)
Recently uploaded
Recently uploaded (20)
PAC 2020 Santorin - Giovanni Paolo Gibilisco
- 1. PERFORMANCE IS NOT A MYTH P E R F O R M A N C E A D V I S O R Y C O U N C I L SANTORINI GREECE FEBRUARY 26 - 27 2020 Continuous Performance Optimization of Java Applications Giovanni Paolo Gibilisco
- 2. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Introduction • Automation and CI have done a great job simplifying deployment and environment provisioning • Performance testing tools are evolving to ease integration • We should take the baton and go further evolving the way we do performance testing and optimization
- 3. P E R F O R M A N C E A D V I S O R Y C O U N C I L Continuous Performance Testing Continuous Performance Optimization Use Case
- 4. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Continuous Performance Testing • This is a hot topic for Performance Engineers • Stephen Townshend – “Preparing for the pipeline” • Alexander Podelko – “Continuous Performance Testing: Myths and Realities” • Bruno Da Silva – “Continuous Performance Innovative Approach” • Amir Rozenberg – “Shifting left performance testing” • Tingting Zong – “Integrate Performance test with DevOps”
- 5. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L A common toolset
- 6. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L A common pipeline • Performance tests usually focus on a subset of components or functionalities • Focus on finding performance regressions Build Unit Test Integration Test End 2 End TestDeploy Performance Test Release Prepare Test Data Run Test Analyze ResultsProvision Load Infrastructure
- 7. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Why stop here? • Each release might change the way application uses resources • Performance Regression make sure you don’t do worst than previous iteration • What about doing better than before? Performance Improvements are left on the table “I’ve changed my data structure, should I switch to another GC type?” “Has my heap usage pattern changed?”
- 8. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Configuration is key • Treated as application code • Has huge impact on performance • Most of the time vendor default is used • Growing Complexity 2006 2009 2013 2016 2019 NumberofConfigurationOptions 847 520 270 50
- 9. P E R F O R M A N C E A D V I S O R Y C O U N C I L Continuous Performance Testing Continuous Performance Optimization Use Case
- 10. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Continuous Performance Optimization • A new kid in the pipeline block • See performance improvements of application changes immediately. Not just regressions • Adapt configuration to new application features and releases Build Unit Test Integration Test End 2 End TestDeploy Performance Test Prepare Test Data Run Test Analyze ResultsProvision Load Infrastructure Optimize Release
- 11. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Why it is so hard? • Configuration Selection • Manual (slow, expertise, trial and error) • One parameter change at a time • Huge configuration space (is random exploration effective?) • Machine Learning comes to aid • Analyze performance test and assign a performance score • Usually manual (slow, need expertise) • Some time a Boolean score based on thresholds (e.g. alarms, gateways) • Challenges: startup, noise, stability, … • Multiple KPIs (Throughput, Latency, Utilization..) Run Test Analyze ResultsPrepare Test Data Optimize
- 12. P E R F O R M A N C E A D V I S O R Y C O U N C I L Continuous Performance Testing Continuous Performance Optimization Use Case
- 13. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L The App • Core business application in charge of searching for flight combinations • “Searcho” is a microservice composed by Tomcat and a JVM application • JVM configuration is written in a file and committed to a Git repo • Gitlab Pipeline takes care of deploying the configuration changes • Kubernetes deployment takes care of restarting the services Searcho Dev Namespace
- 14. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L The Goal Maximize Searcho Transactions Per Second … complying with the following constraints (5 mins evaluation window): • stability: TPS std. dev. within the window must be < 3 • response time: avg. response times within the window must be < 4 sec • errors • http error rate within the window must be < 1% • log errors within the windows must be < 10
- 15. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Optimization Scope Which is the best Garbage Collection (GC) algo for this workload? • Baseline: reference value of Searcho with a manually tuned JVM configuration • G1: study on 12 JVM parameters using G1 GC algorithm • Parallel: optimized 11 JVM parameters using Parallel GC algorithm 11.03 13.6 11.69 Baseline Parallel G1 Baseline Parallel G1 +23% +5%
- 16. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Peak throughput Best window that satisfies all the constraints in baseline experiment: 11.03 TPS Best window that satisfies all the constraints in best experiment: 12.61 TPS
- 17. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Lower response time Baseline: 7.5+ sec @63 VU Best: 5.5 sec @63 VU
- 18. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Lower GC time Baseline: 200 ms @63 VU Best: 40 ms @63 VU
- 19. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Lower throttling Baseline: 1.7+ sec @63 VU Best: 0.5 sec @63 VU
- 20. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Higher memory efficiency • Overall heap utilization has been reduced • Compilation parameters optimized to reduce CPU consumption • Optimized heap inner space • Startup memory allocation to avoid noisy neighbors • Increased GC parallelism to reduce pauses Configuration Peak TPS Heap Size (MB) TPS/Heap Serchio default 11.03 6656 0.0017 High TPS 13.61 5714 0.0023 (+35%) Efficiency 12.31 4152 0.0030 (+76%)
- 21. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L A Stability study Can we serve traffic without degrading quality when workload is increasing? • … complying with the following constraints: • response time: avg. response times within the window has to be < 7.5 sec • throughput: TPS within the window must be > 10 • Baseline: reference value of Searchio with tuned configuration • Tomcat: optimized 3 tomcat parameters related to threading and queuing
- 22. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Keep serving traffic Baseline: 100% error from 90 VU Best: 13.61 TPS @ 100 VU Same behavior up to the breaking point
- 23. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Reduce response time degradation Baseline: 11+ sec @ 90 VU Best: 7.5 sec @ 100 VU Same behavior up to the breaking point
- 24. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Lower GC time Baseline: 150 +ms @ 90 VU Best: 20 ms @ 100 VU Same behavior up to the breaking point
- 25. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Lower throttling Baseline: 3 ms @ 90 VU Best: 0.2 ms @ 100 VU Same behavior up to the breaking point
- 26. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Takeaways • Automation in CI is great but is just a starting point • Continuous Performance Testing allows to avoid regressions and let us sleep well • Manual Optimization is hard, tedious and ages as faster as the software evolves • Continuous Performance Optimization let us find performance improvements due to application changes