WE are Doing it Wrong - Dmitry Sharkov
•
3 likes•2,834 views
As companies mature their software development practices, automated acceptance-level testing is becoming more commonplace. In particular, Cucumber and its Gherkin-based equivalents are enjoying widespread use. Through observing and facilitating the adoption and implementation of Cucumber test suites, I have found ways in which the technology has helped teams greatly, but I have also found ways in which it hindered them. I realized that Cucumber and its kin are appropriate tools in fewer situations than the ones in which they are currently employed. In other words, many teams that use such frameworks need to reevaluate whether they are right for the job, and perhaps replace them. I invite all involved in automated acceptance testing to attend as I try to build a compelling case for this notion.
Report
Share
WE are Doing it Wrong - Dmitry Sharkov
- 1. Cucumber WE are Doing it Wrong name = 'Dmitry Sharkov' twitter = "@#{name.gsub(' ','')}" company = 'CareWorks Tech'
- 3. No. But it is frequently used inappropriately and ends up not as valuable as expected. http://whynne.deviantart.com/
- 4. What we will talk about: Cucumber's value proposition Challenges that I have seen teams experience with it Various developers hating on Cucumber Cucumber's developer bias High cost of Cucumber as a functional spec The compromises made Alternatives to Cucumber Selecting the right tool for the situation
- 5. • Devs understanding requirements is necessary • Business-legible specification is valuable • Acceptance-level testing is valuable • Automated testing is key to agility • Traceability from specification to tests is valuable • Green Arrow is basically a "budget" Batman • Capturing preconditions in specs is valuable Let's agree that…
- 6. Sounds a lot like… www.effenvodka.com/ and many teams struggle with it.
- 7. Cukes gone bad Business does not contribute to or consume Feature (Gherkin) files. Developers write and use them. The Gherkin produced is poor and difficult to elegantly map to code. Tests are taking up almost as much time/money as implementation code. Test suites take too long to complete. Test-first approach doesn't last long. http://plantsvszombies.wikia.com/wiki/File:Nuclear_Cucumber.png
- 8. Are devs writing the Gherkin? Business creates requirements with JIRA or a wiki and shows little interest in Gherkin. Devs write it since they're using Cucumber. Or the business generates poor Gherkin. Devs re-write it since they're using Cucumber. This can lead to frustration and wasted time.
- 9. Does the Cucumber test suite take too long to run outside of a CI grid? Does it not get run as code is written, prior to integration? Are the tests too slow in CI to provide a good feedback loop? Regressions are being caught later. http://pokemon.wikia.com/wiki/Slowpoke
- 10. Does Cucumber feel too costly? Estimates are inflated due to Cucumber. Developers see it as a burden and don't feel a positive R.O.I. Do developers often joke about writing lots of test code to test a minor change? Low velocity, low morale. Cucumber is at risk.
- 11. Too expensive, too difficult, too slow. Over-using Cucumber. Not capturing as much as possible in unit and integration tests. Possibly over-testing altogether. Look at what questions remain unanswered and answer them in the simplest, fastest way possible. http://i.stack.imgur.com/U1mq6.jpg
- 12. Business and development having trouble with the Gherkin IS unique to Cucumber. Why? Problems of cost, speed, and complexity are not unique to Cucumber.
- 13. Cucumber is actually NOT great at this • Devs understanding requirements is necessary • Business-legible specification is valuable • Acceptance-level testing is valuable • Automated testing is key to agility • Traceability from specification to tests is valuable • Green Arrow is basically a "budget" Batman • Capturing preconditions in specs is valuable
- 14. By the way, this isn't news. It's easy to find developers critiquing Cucumber online. https://www.flickr.com/photos/sabrewer_1/3767863496/
- 15. –James Shore (author of The Art Of Agile Development) http://www.jamesshore.com/Blog/The-Problems-With-Acceptance-Testing.html “These two problems--that customers don't participate, which eliminates the purpose of acceptance testing, and that they create a significant maintenance burden, means that acceptance testing isn't worth the cost. I no longer use it or recommend it.”
- 16. –Kevin Liddle http://blog.8thlight.com/kevin-liddle/2013/09/18/a-case-against-cucumber.html “In theory, this is a valuable thing, a bridge between the divergent worlds of developers and managers. In practice, however, I’ve never seen Cucumber used this way… Quit writing [Gherkin] unless you can honestly say that there is someone reading them who would not understand pure Ruby.”
- 17. –David Hansson (creator of RoR) https://signalvnoise.com/posts/3159-testing-like-the-tsa “Don’t use Cucumber unless you live in the magic kingdom of non-programmers-writing-tests (and send me a bottle of fairy dust if you’re there!)”
- 18. –Robert "Uncle Bob" Martin http://blog.8thlight.com/uncle-bob/2013/09/26/AT-FAIL.html “...the programmers… decided to write the acceptance tests themselves…. ...the point was to get the business to provide a formal specification of the system so that the programmers could understand it. What… is the point of having the programmers write the formal specification so that the programmers can then understand it? But then... the Cucumber folks decided to turn Cucumber into a programmer tool! …They made rampant use of regular expressions. They did everything they could to make it easy for programmers to write these tests.” Never offers a solution, but does clarify the problem.
- 19. Strategies are at odds. Art by Jim Lee, Alex Sinclair, Scott Williams Steps should be business-oriented and focused. Steps are reusable and support composition. Cucumber is helpful if you want to do this
- 20. Cucumber is a great developer-oriented tool. Regular expressions Nested steps Hooks and Tags Transforms http://cucumber.gigidigi.com/ Reuse, maintainability, general "code quality" Harder for non-devs to write the Gherkin.
- 21. Using Cucumber's developer-friendly features can yield a poor functional spec: @exchangeUser Scenario: user resets password Given I am not logged in When I go to the login page And I click on the forgot password link And I am on the forgot password page And I enter my email address And click the reset password button Then I get an email with a reset password link When I click on the link in the email Then I go to the reset password page And I enter a new password …
- 22. Scenario: user resets password Given I forgot my password When I reset it with an email Then I can log in with the new password A business-oriented functional spec reads better but may have other issues. • Steps are rarely reusable. • Steps can turn out to be no-ops. • Important interactions are hidden. • Maintainability is not actually improved, either. (Code is likely less modular and more repetitive.)
- 23. Write legible, focused business specification and not much of the Cucumber framework is taken advantage of. Cucumber forces a compromise if we want non-technical people to author the spec.
- 24. Gherkin can make for great specification. But Cucumber's stepwise automation of Gherkin is a double-edged sword. Gherkin != Cucumber
- 25. Gherkin != Cucumber Gherkin's value for a specification is to force consideration of preconditions. Cucumber's value is to make it a living, executable spec. Costs:value curves are linear vs logarithmic. Cucumber needs Gherkin. But not vice-versa. http://thebigfatnoodle.com/top-nosh/the-gherkins-4030-restaurant-and-bar/
- 26. I have always argued in favor of But at times I have felt the tight coupling between steps and code to be a hindrance.
- 28. Not quite ready for prime time… …but may become a useful option.
- 29. An updated, functioning version is in development. (quick demo)
- 30. Let's revisit what our NEEDS are. Does Cucumber help meet them? Does Gherkin? Might something else?
- 31. • Devs understanding requirements is necessary • Business-legible specification is valuable • Acceptance-level testing is valuable • Automated testing is key to agility • Traceability from specification to tests is valuable • Capturing preconditions in specs is valuable
- 32. Developers understanding requirements Gherkin done well is best due to specification by example. Done poorly it is probably worse than a simple list of acceptance criteria. JIRA (or the like) can link to supporting docs (e.g. wireframes). Whatever tool helps facilitate conversation between devs and the business (or BA) is the one to use.
- 33. Business-legible specification Cucumber allows for very legible specs with a bit of work. rspec-gherkin allows for them with a bit less work (steps are not mapped directly). RSpec can produce legible reports but they will probably be much less precise. JIRA puts no constraints on a BA so you can have maximum legibility but absolutely no coupling to tests.
- 34. Automated, acceptance-level testing Cucumber can do it. RSpec can do it. rspec-gherkin can do it. Any test runner that can interface with an app driver can. If Gherkin isn't valued, then anything but RSpec forces extra work. You need good test engineers to get it done in any case.
- 35. Traceability from Specification to Test Cucumber does it well but mapping of individual steps puts a burden on spec authors. rspec-gherkin has less strict translation at lower cost. RSpec does not capture specification as well.
- 36. Capturing preconditions in specs Gherkin is just about the only way to guarantee this occurs.
- 37. Seems like Gherkin has clear advantages. But if the business does not collaborate in its creation, and does not consume it otherwise, then it is mostly waste. And it is more difficult to write if steps are mapped to code rather than whole scenarios.
- 38. Is rspec-gherkin superior to Cucumber? • No coupling of individual steps to code blocks. • Less of a burden on Gherkin authors. • Less of a burden of communication and collaboration. • Particularly in cases where legacy software has no tests, no spec, and no dedicated business stakeholder, rspec-gherkin might work well.
- 39. Is there a case where Cucumber is clearly superior? Yes… when there are many manual testers who cannot transition to programming. http://blog.eat24hours.com/12-hidden-meanings-of-popular-food-phrases/
- 40. What I wish we had… A platform connected to multiple test runners that provided an option to have tight coupling (of steps via Cucumber), loose coupling (of Scenarios via rspec-gherkin), or meta reporting (from lower-level tests) on a per-Scenario basis.
- 41. If the business does not consume Gherkin, just use RSpec. If the business wants to use Gherkin, consider Cucumber or rspec-gherkin based on circumstances. Maybe we need a new tool altogether. If it ain't broke, don't fix it. :-/ CONCLUSIONS