SlideShare a Scribd company logo

Automatic testing and quality assurance for WordPress plugins and themes

•
•
Otto Kekäläinen
Otto Kekäläinen

Talk given at WP Helsinki Meetup 7.11.2018 See also: * https://developer.wordpress.org/themes * https://developer.wordpress.org/plugins * https://travis-ci.org/Seravo * https://seravo.com/blog/coding-wordpress-in-style-with-phpcs/

Related slideshows

Php Power Tools
Php Power ToolsPhp Power Tools
Php Power Tools
 
Why it's dangerous to turn off automatic updates and here's how to do it
Why it's dangerous to turn off automatic updates and here's how to do itWhy it's dangerous to turn off automatic updates and here's how to do it
Why it's dangerous to turn off automatic updates and here's how to do it
 
Improving WordPress performance (xdebug and profiling)
Improving WordPress performance (xdebug and profiling)Improving WordPress performance (xdebug and profiling)
Improving WordPress performance (xdebug and profiling)
 
1 of 38
Download to read offline
Automatic testing and quality assurance for WordPress plugins and themes WP Helsinki Meetup 7.11.2018 Otto Kekäläinen @ottokekalainen WP-palvelu.fi / Seravo.com
● WP-palvelu.fi – WordPress hosting and upkeep ● CEO, sysadmin and developer ● Linux and open source advocate ● Contributed to WordPress Core, fi and sv translations, Linux, Docker, Nginx, Redis, MariaDB… ● Twitter:@ottokekalainen Otto Kekäläinen
Enterprise grade hosting and upkeep for WordPress
FIRST THINGS FIRST ● Before you write any WordPress theme or plugin code, please read up on the basics at: ○ developer.wordpress.org/themes ○ developer.wordpress.org/plugins
CHALLENGE: EVERYBODY WANTS QUALITY Developers just don’t have enough time or customers budget
Solution: Automatic ~ zero cost
DO EVERYTHING THAT CAN BE AUTOMATED ● scan code to find errors ○ static analysis ● run code to find errors ○ unit and integration tests
TOOLS 1. What can test PHP code 2. What can automate tests
WHAT TO DO ABOUT PHP CODE ● PHP Code Sniffer ● PHP unit tests ● PhantomJS Headless Chrome integration tests ● performance ○ execution time ○ memory usage
PHP CODE SNIFFER
PHPCS
PHPCFB AND GIT CITOOL
phpcs.xml <?xml version="1.0"?> <ruleset name="Seravo"> <arg value="p"/> <file>.</file> <rule ref="Squiz.PHP.CommentedOutCode"/> <rule ref="Squiz.WhiteSpace.SuperfluousWhitespace"/> <rule ref="Generic.CodeAnalysis.UnusedFunctionParameter"/> <rule ref="Generic.Commenting.Todo"/> <rule ref="Generic.ControlStructures.InlineControlStructure"/> <rule ref="WordPress-Extra"> <exclude name="Generic.WhiteSpace.DisallowSpaceIndent"/> <exclude name="Generic.WhiteSpace.ScopeIndent"/> <exclude name="WordPress.WhiteSpace.PrecisionAlignment.Found" /> <exclude name="WordPress.PHP.YodaConditions" /> </rule> </ruleset>
phpcs --standard=Security FILE: tools/gapi.php --------------------------------------------------------------------- FOUND 1 ERROR AND 8 WARNINGS AFFECTING 8 LINES --------------------------------------------------------------------- 35 | WARNING | Possible RFI detected with GADWP_DIR on include_once 51 | WARNING | Function array_map() that supports callback detected 148 | WARNING | Possible XSS detected with esc_url on echo 152 | WARNING | Possible XSS detected with __ on echo 156 | WARNING | Possible XSS detected with _e on echo 307 | WARNING | Crypto function crc32 used. 767 | WARNING | Function array_map() that supports callback --------------------------------------------------------------------- More tips at seravo.com/coding-wordpress-in-style-with-phpcs
PHP UNIT TESTS assertEquals(a, b) Lots of PHP Unit test examples in WordPress Core source
ACCEPTANCE TESTS Codeception PHP framework + Headless Chromium
Example test code <?php class ExampleCest { /** * Open front page (/) **/ public function openFrontPage(AcceptanceTester $I) { $I->amOnPage('/'); $I->checkBrowserConsole(); $I->see('WordPress'); } } Read more at seravo.com/docs/tests/ng-integration-tests
VISUAL REGRESSION TESTS $ gm compare -highlight-style assign -highlight-color purple -file diff.png *.png
VISUAL REGRESSION TESTS $ gm compare -verbose -metric mse *.png Image Difference (MeanSquaredError): Normalized Absolute ============ ========== Red: 0.0319159868 8.1 Green: 0.0251841368 6.4 Blue: 0.0278537225 7.1 Opacity: 0.0000000000 0.0 Total: 0.0212384615 5.4
Where do you draw the line between acceptable changes and failures/regressions?
HOW TO AUTOMATE ● git pre-commit hook ○ local tests ● git receive hook on a remote server ○ Github + Travis-CI ○ Gitlab + Gitlab-CI ○ Bitbucket ○ etc..
GIT PRE-COMMIT HOOK IN ACTION
Example .git/hooks/pre-commit # Loop all files that are about to be committed (diff of git head and staged) echo "==> Checking syntax errors..." for FILE in $(git diff --cached --name-only); do resource="$REPO_DIR/$FILE" ## # Test PHP syntax for all changed *.php and *.module files ## if [[ "$FILE" =~ ^.+(php|module)$ ]]; then if [[ -f $resource ]]; then phpcs "$resource" 1> /dev/null if [ $? -ne 0 ]; then errors+=("PHP syntax Error: $FILE") fi fi fi done See code at seravo.com/coding-wordpress-in-style-with-phpcs
TRAVIS-CI IN ACTION
TRAVIS-CI IN ACTION
TRAVIS-CI CHECKING EVERY COMMIT
..AND PULL REQUESTS!
NOTIFICATION EMAILS THAT CAN’T GO UNNOTICED
Example .travis.yml before_install: - if [[ "$SNIFF" == "1" ]]; then git clone … script: # Syntax check all php files and fail for any error text in STDERR - '! find . -type f -name "*.php" -exec php -d error_reporting=32767 -l {} ; 2>&1 >&- | grep "^"' # More extensive PHP Style Check - if [[ "$SNIFF" == "1" ]]; then $PHPCS_DIR/bin/phpcs -i; $PHPCS_DIR/bin/phpcs --standard=phpcs.xml; fi - phpunit
Live examples github.com/Seravo/seravo-plugin github.com/Seravo/linux-tuki.fi travis-ci.org/Seravo
FREE FOR OPEN SOURCE CODE SERVICES ● circleci.com ● cocodacy.com ● codeclimate.com ● codeship.com ● coveralls.io ● coverity.com ● sourceclear.com ● travis-ci.org (.com for private repos) Listed in alphabetic order, no preference.
Measure execution time and memory echo ""; $ for x in {1..20} do curl -s http://localhost | grep "Measurements" done </div><div id=Test with dummy data ● While developing a site, load lots of dummy data into it so you can test how your site looks and performs with 100, 1000 or 100 000 posts. ● Basic: Import themeunittestdata.wordpress.xml ○ codex.wordpress.org/Theme_Unit_Test ● More data: wp post generate ○ curl http://loripsum.net/api/5 | wp post generate --post_content --count=10 ● More realism: wp-cli-fixtures ○ github.com/nlemoine/wp-cli-fixtures
WordPress plugins have a reputation of low quality. Help us prove them wrong. Start using automatic quality testing!
Hopefully automatic quality testing will be integrated into the WordPress.org plugin directory in the future. See make.wordpress.org/tide
Extra tip: use the WP plugin boiler plate to start with: wppb.io WP Theme starter example: github.com/aucor/aucor-starter
This presentation was about plugin and theme development. How about testing a real WordPress site to ensure updates don’t break it? Let Seravo handle updates of production sites for you. See our hosting and upkeep service at Seravo.com
THANK YOU! KIITOS! @Seravo @SeravoFi @ottokekalainen

More Related Content

Automatic testing and quality assurance for WordPress plugins and themes

  • 1. Automatic testing and quality assurance for WordPress plugins and themes WP Helsinki Meetup 7.11.2018 Otto Kekäläinen @ottokekalainen WP-palvelu.fi / Seravo.com
  • 2. ● WP-palvelu.fi – WordPress hosting and upkeep ● CEO, sysadmin and developer ● Linux and open source advocate ● Contributed to WordPress Core, fi and sv translations, Linux, Docker, Nginx, Redis, MariaDB… ● Twitter:@ottokekalainen Otto Kekäläinen
  • 3. Enterprise grade hosting and upkeep for WordPress
  • 4. FIRST THINGS FIRST ● Before you write any WordPress theme or plugin code, please read up on the basics at: ○ developer.wordpress.org/themes ○ developer.wordpress.org/plugins
  • 5. CHALLENGE: EVERYBODY WANTS QUALITY Developers just don’t have enough time or customers budget
  • 7. DO EVERYTHING THAT CAN BE AUTOMATED ● scan code to find errors ○ static analysis ● run code to find errors ○ unit and integration tests
  • 8. TOOLS 1. What can test PHP code 2. What can automate tests
  • 9. WHAT TO DO ABOUT PHP CODE ● PHP Code Sniffer ● PHP unit tests ● PhantomJS Headless Chrome integration tests ● performance ○ execution time ○ memory usage
  • 11. PHPCS
  • 12. PHPCFB AND GIT CITOOL
  • 13. phpcs.xml <?xml version="1.0"?> <ruleset name="Seravo"> <!-- show progress --> <arg value="p"/> <!-- check current and all subfolders if no file parameter given --> <file>.</file> <rule ref="Squiz.PHP.CommentedOutCode"/> <rule ref="Squiz.WhiteSpace.SuperfluousWhitespace"/> <rule ref="Generic.CodeAnalysis.UnusedFunctionParameter"/> <rule ref="Generic.Commenting.Todo"/> <rule ref="Generic.ControlStructures.InlineControlStructure"/> <rule ref="WordPress-Extra"> <exclude name="Generic.WhiteSpace.DisallowSpaceIndent"/> <exclude name="Generic.WhiteSpace.ScopeIndent"/> <exclude name="WordPress.WhiteSpace.PrecisionAlignment.Found" /> <exclude name="WordPress.PHP.YodaConditions" /> </rule> </ruleset>
  • 14. phpcs --standard=Security FILE: tools/gapi.php --------------------------------------------------------------------- FOUND 1 ERROR AND 8 WARNINGS AFFECTING 8 LINES --------------------------------------------------------------------- 35 | WARNING | Possible RFI detected with GADWP_DIR on include_once 51 | WARNING | Function array_map() that supports callback detected 148 | WARNING | Possible XSS detected with esc_url on echo 152 | WARNING | Possible XSS detected with __ on echo 156 | WARNING | Possible XSS detected with _e on echo 307 | WARNING | Crypto function crc32 used. 767 | WARNING | Function array_map() that supports callback --------------------------------------------------------------------- More tips at seravo.com/coding-wordpress-in-style-with-phpcs
  • 15. PHP UNIT TESTS assertEquals(a, b) Lots of PHP Unit test examples in WordPress Core source
  • 17. Example test code <?php class ExampleCest { /** * Open front page (/) **/ public function openFrontPage(AcceptanceTester $I) { $I->amOnPage('/'); $I->checkBrowserConsole(); $I->see('WordPress'); } } Read more at seravo.com/docs/tests/ng-integration-tests
  • 18. VISUAL REGRESSION TESTS $ gm compare -highlight-style assign -highlight-color purple -file diff.png *.png
  • 19. VISUAL REGRESSION TESTS $ gm compare -verbose -metric mse *.png Image Difference (MeanSquaredError): Normalized Absolute ============ ========== Red: 0.0319159868 8.1 Green: 0.0251841368 6.4 Blue: 0.0278537225 7.1 Opacity: 0.0000000000 0.0 Total: 0.0212384615 5.4
  • 20. Where do you draw the line between acceptable changes and failures/regressions?
  • 21. HOW TO AUTOMATE ● git pre-commit hook ○ local tests ● git receive hook on a remote server ○ Github + Travis-CI ○ Gitlab + Gitlab-CI ○ Bitbucket ○ etc..
  • 22. GIT PRE-COMMIT HOOK IN ACTION
  • 23. Example .git/hooks/pre-commit # Loop all files that are about to be committed (diff of git head and staged) echo "==> Checking syntax errors..." for FILE in $(git diff --cached --name-only); do resource="$REPO_DIR/$FILE" ## # Test PHP syntax for all changed *.php and *.module files ## if [[ "$FILE" =~ ^.+(php|module)$ ]]; then if [[ -f $resource ]]; then phpcs "$resource" 1> /dev/null if [ $? -ne 0 ]; then errors+=("PHP syntax Error: $FILE") fi fi fi done See code at seravo.com/coding-wordpress-in-style-with-phpcs
  • 29. Example .travis.yml before_install: - if [[ "$SNIFF" == "1" ]]; then git clone … script: # Syntax check all php files and fail for any error text in STDERR - '! find . -type f -name "*.php" -exec php -d error_reporting=32767 -l {} ; 2>&1 >&- | grep "^"' # More extensive PHP Style Check - if [[ "$SNIFF" == "1" ]]; then $PHPCS_DIR/bin/phpcs -i; $PHPCS_DIR/bin/phpcs --standard=phpcs.xml; fi - phpunit
  • 31. FREE FOR OPEN SOURCE CODE SERVICES ● circleci.com ● cocodacy.com ● codeclimate.com ● codeship.com ● coveralls.io ● coverity.com ● sourceclear.com ● travis-ci.org (.com for private repos) Listed in alphabetic order, no preference.
  • 32. Measure execution time and memory echo "<!-- Measurements: "; echo memory_get_usage(); echo " - "; echo (microtime(true) - $_SERVER["REQUEST_TIME_FLOAT"]); echo " -->"; $ for x in {1..20} do curl -s http://localhost | grep "Measurements" done <!-- Measurements: 5761720 - 0.2341411113739 <!-- Measurements: 5761720 - 0.24964690208435 <!-- Measurements: 5761704 - 0.25908708572388 <!-- Measurements: 5761720 - 0.23540115356445 ...
  • 33. Test with dummy data ● While developing a site, load lots of dummy data into it so you can test how your site looks and performs with 100, 1000 or 100 000 posts. ● Basic: Import themeunittestdata.wordpress.xml ○ codex.wordpress.org/Theme_Unit_Test ● More data: wp post generate ○ curl http://loripsum.net/api/5 | wp post generate --post_content --count=10 ● More realism: wp-cli-fixtures ○ github.com/nlemoine/wp-cli-fixtures
  • 34. WordPress plugins have a reputation of low quality. Help us prove them wrong. Start using automatic quality testing!
  • 35. Hopefully automatic quality testing will be integrated into the WordPress.org plugin directory in the future. See make.wordpress.org/tide
  • 36. Extra tip: use the WP plugin boiler plate to start with: wppb.io WP Theme starter example: github.com/aucor/aucor-starter
  • 37. This presentation was about plugin and theme development. How about testing a real WordPress site to ensure updates don’t break it? Let Seravo handle updates of production sites for you. See our hosting and upkeep service at Seravo.com