Q2.12: Benchmarking Techniques
•
2 likes•551 views
This document summarizes benchmarking techniques for evaluating software and hardware performance. It discusses the importance of using relevant benchmarks, accurately measuring performance with timers, accounting for external influences, running tests repeatedly to obtain statistics, and analyzing results using statistical methods like standard deviation and t-tests to determine significance of differences. The benchmarks, tools, and processes used by Linaro for benchmarking are also outlined.
Report
Share
Q2.12: Benchmarking Techniques
- 1. Benchmarking Techniques Michael Hope <michael.hope@linaro.org> bzr branch lp:~michaelh1/+junk/benchmarking-techniques r6
- 4. 4 Picking relevant benchmarks: Profile Workload Features We use SPEC 2000 and EEMBC We'd like shareable benchmarks
- 5. 5 Test Platform Build / test / benchmark via Linux / web / commodity hardwar
- 6. 6 Measuring
- 7. 7 Realtime timers See 'man clock_gettime'
- 8. 8 <comparison of different timers> See https://wiki.linaro.org/WorkingGroups/ToolChain/Benchmarks/TimerAc
- 9. 9
- 10. 10
- 12. 12 Other features to be wary of scheduler governor cpuidle, power management, thermal limiting SMP bugs, like core lockdown NEON startup See “Understanding the Linux Kernel” ch10: http://oreilly.com/catalog/linuxkernel/chapter/ch10.html
- 13. 13 Putting things together and running We use: timer built into the app run five times collect everything post process
- 14. 14 Statistics
- 15. 15
- 16. 16
- 17. 17 Standard deviation Dispersion / coefficient of variance t-scores t-test
- 18. 18 t= ̄X 1 − ̄X 2 √ s1 2 N 1 + s2 2 N 2 See http://en.wikipedia.org/wiki/Welch%27s_t_
- 19. 19 Compiler Mean Std CV gcc-4.6.2 1.00 134u 134u gcc-linaro-4.6-2011.11 4.25 5035u 1178u t = 30,300
- 20. 21 Variant Mean Std CV Plain 1.00 320u 320u With SMS 1.01 430u 426u t = 118
- 21. 22 Variant Mean Std CV Plain 1.00 320u 320u With vectoriser 1.001 404u 404u t = 8.27 - significant
- 22. 23 Our tools perf difftest betters tabulate Python LibreOffice! Other statistical tools like scipy.stat, R, Judge, and ministat