PushToTest TestMaker 6.5 Open Source Test Design Document
•
1 like•1,166 views
PushToTest TestMaker version 6.5 product design document for a major feature enhancement. Contains user interface definitions, product roadmap, and feature requirements. Please comment on this to improve TestMaker.
Report
Share
PushToTest TestMaker 6.5 Open Source Test Design Document
- 1. Collaborative Results, Resource Repository, UI Enhancements TestMaker 6.5 Plan Frank Cohen, CEO (408) 871-0122 fcohen@pushtotest.com November 8, 2011 Intellectual Property of PushToTest. (c) 2011 All rights reserved. Note: All trademarks mentioned in this document are the trademarks of their respective holders.
- 2. Open Source Test Automation Intellectual Property of PushToTest. (c) 2011 All rights reserved. Product Roadmap TestMaker 6.0 TestMaker 6.1 TestMaker 6.5 TestMaker 7 2011 2011 2011 2012 • Improved Test • Tabular Results Charts • Easier Test Controller • TestCards Phase 2 Operation Speed • Flex 4 In-Browser • Reports • Web Reports • Jasper Business Recording • Collaborative Charts • Web Controller Intelligence Reporting • TestMaker JAT for • Live Charts for Drill • Test Management • Test Object Designer Oracle Applications Downs • Rally • Record/Playback • Citrix Applications • Test Object Repository • TestLink • Firefox, IE, Safari, • TestCards Phase 1 • Share Test Chrome, Opera • HP QC • Rally Integration Resources • Open Source • Zephyr • Web Reports • Search Alternative to • QMetry HP QTP • Web Controller • Access Control • Adds Sahi, Flex • Bug Fixes • LDAP/Rally • TestCards 1 • Data and Object Enabled • HtmlUnit Viewer
- 3. 3 Open Source Test Automation Intellectual Property of PushToTest. (c) 2011 All rights reserved. TestMaker 6.5 ‣ Resource Storage in the Repository ‣ Access Control for users and LDAP integration ‣ Search ‣ Reports ‣ Live Reports ‣ Collaborative Reports ‣ Easier Test Controller ‣ Streaming Results using Snapshot Protocol
- 4. 4 Open Source Test Automation Intellectual Property of PushToTest. (c) 2011 All rights reserved. Needs Research for 6.5 ‣ TestMaker to restore Windows and IE proxy on quit ‣ Console Connects To TestNodes (Cloud) Through Proxy ‣ URL to Web-based TestMaker Test Results Automatically Saved To Rally Test Case ‣ L7 and HPOpenView reports gathered automatically
- 5. 5 Open Source Test Automation Intellectual Property of PushToTest. (c) 2011 All rights reserved. Not In TestMaker 6.5 ‣ Full Web Interface for TestMaker ‣ Test ManagementFailed Previous Test Operations ‣ Rerun Tests of Zephyr, Quality Center Integration ‣ TestLink, QMetry,Unique DPL v2) ‣ Colossus“Push” Bundle Deployment, Use Resources Instead DPL (aka ‣ DynamicTestNode Sahi Proxy SSL Certificate Registration in ‣ Remote Community ‣ Open Designer,view “How Are We Doing” see Editor too ‣ No dashboard has a custom status display (icon and text) settable ‣ Each Controller step context by API from a test ‣ Graphical view to edit and manage RDBMS for DPL content
- 6. 6 Open Source Test Automation Intellectual Property of PushToTest. (c) 2011 All rights reserved. Resource Repository ‣ File System or Repository ‣ Editor, TestMaker Object Designer, Console ‣ Open, Save, Save As Open Test Resource Save Test Resource Repositories Repositories Local Repository Local Repository QA Unit 1 Repository QA Unit 1 Repository / Last Modified Creator / Last Modified Creator Project X June 12, 2011 6:05 pm mcohen 3rd Party Connector Project June 12, 2011 6:12 pm mcohen Project X June 12, 2011 6:05 pm mcohen TestScenarios June 12, 2011 12:05 pm mcohen 3rd Party Connector Project June 12, 2011 6:12 pm mcohen Load Test 103 Scenario June 12, 2011 1:05 pm mcohen TestScenarios June 12, 2011 12:05 pm mcohen Project Y June 12, 2011 3:05 pm mcohen Load Test 103 Scenario June 12, 2011 1:05 pm mcohen Project Z June 12, 2011 5:05 pm mcohen Project Y June 12, 2011 3:05 pm mcohen Project Z June 12, 2011 5:05 pm mcohen Load Test 133 Scenario Cancel Open Show Versions Cancel Save Show Versions
- 7. 7 Open Source Test Automation Intellectual Property of PushToTest. (c) 2011 All rights reserved. Resource Repository Search ‣ Search File Names ‣ Search TestMaker Object Designer Meta Data ‣ Name, Creator, Links, Version, Types, Operations, Goals ‣ Object Repository objects: Story, Form, Object ‣ Object name, Type, Accessor, Comment ‣ Search Editor Meta Data ‣ Author Name, Creator, Links, Versions, Categories
- 8. 8 Open Source Test Automation Intellectual Property of PushToTest. (c) 2011 All rights reserved. Access Control ‣ TestCard Viewable by Anonymous, Role, User Name ‣ TestScenarioRole, User Name Stop, Pause, Create, Delete by Anonymous, View, Change, Run, ‣ Repository Read, Change, Create DB, Create Contents, Delete by Anonymous, Role, User Name ‣ Role Create, Change, Delete by Anonymous, Role, User Name ‣ Reports Create, Change, Delete by Anonymous, Role, User Name
- 9. 9 Open Source Test Automation Intellectual Property of PushToTest. (c) 2011 All rights reserved. SSO Option with Rally ‣ Optionally Uses Rally LDAP for User Authentication PushToTest TestCards - Best Buy help Welcome To TestMaker TestCards TestMaker Enterprise 6.0.1 User ID fcohen Licensed for 50 virtual users and 10 service monitors Password ••••••••• Sign-in Options Sign-in Using Rally
- 10. 10 Open Source Test Automation Intellectual Property of PushToTest. (c) 2011 All rights reserved. Enhanced Controller ‣ Web (for TCS 1) and SWING ‣ Reports Button -> New Reports Ajax Load Test ‣ Tabs Expand To New Window 513 40 Seconds Remaining Executing iteration: 1 Users * 1 513 Success 12.67 % Failure 65 Failed Instant Results TestNode Status Local TestNode Log Local TestNode Error Log Real Time Scalability Index Chart 6 5 4 3 2 1 2.5 TPS 4.6 TPS 1 TPS 0 1 Users * 1 2 Users * 1 4 Users * 1 213 Success 475 Success 513 Success 5 Failed 35 Failed 65 Failed .02% Failure 7.36% Failure 12.67% Failure Reports Configuration Logs
- 11. Open Source Test Automation Intellectual Property of PushToTest. (c) 2011 All rights reserved. Next let’s talk about Reports Design ‣ Resource Plan ‣ User Interface Design ‣ Required Changes to TestMaker 6.0 11
- 12. 12 Open Source Test Automation Intellectual Property of PushToTest. (c) 2011 All rights reserved. Reports Summary ‣ User Interface Design (Tomas) ‣ Reports - A report is a collection of charts ‣ Charts - Athe Transaction of the test results repository. For example, graphical view Distribution chart ‣ Live Charts - A user interface to see custom step logging data ‣ ScriptRunner (EOP) Enhancements (Luis Carlos) ‣ DesignerScript -commandtoinstore screenshots, DOM after executing each Options a .ds file ‣ soapUI - Options to save each TestCase steps request & response ‣ Java - Options to save request, response, stack trace
- 13. 13 Open Source Test Automation Intellectual Property of PushToTest. (c) 2011 All rights reserved. Required Changes ‣ PTTStepListener and Repository Schema ‣ StepRequest (String, may contain CR&LF characters) ‣ StepResponse (Blob, may contain CR&LF characters) ‣ StepStackTrace (String, up to 10K characters) ‣ StepExceptionName (String) ‣ Screenshot (Blob) ‣ TestScenario Schema Changes ‣ Define Report, Charts, Drill Downs
- 14. 14 Open Source Test Automation Intellectual Property of PushToTest. (c) 2011 All rights reserved. Optional Enhancements ‣ Caching of Jasper generated charts to avoid reprocessing time
- 15. 15 Open Source Test Automation Intellectual Property of PushToTest. (c) 2011 All rights reserved. Run A Report PushToTest TestMaker 6 - Best Buy help TestMaker Results Repository Run Report Edit Report Delete Report New Report TestMaker Enterprise 6.0.1 Licensed for 50 virtual users and 10 service monitors Reports s Weekly Scalability Report img TSH Working Team 2 s Load Test for Mods To 3rd Party Gov img Performance after changes s Functional Test for Geek Squad Reg img Testing new registration process
- 16. 16 Open Source Test Automation Intellectual Property of PushToTest. (c) 2011 All rights reserved. Report Appears In Window PushToTest TestMaker 6 - Weekly Scalability Report - Created on September 10, 2011 Weekly Scalability Report Created on September 10, 2011 Transaction Distribution Chart
- 17. 17 Open Source Test Automation Intellectual Property of PushToTest. (c) 2011 All rights reserved. New/Edit A Report PushToTest TestMaker 6 - Best Buy help TestMaker Results Repository TestMaker Enterprise 6.0.1 New Report Licensed for 50 virtual users and 10 service monitors Report Name: Cancel Next
- 18. 18 Open Source Test Automation Intellectual Property of PushToTest. (c) 2011 All rights reserved. Add Charts To The Report PushToTest TestMaker 6 - Best Buy help TestMaker Results Repository Report Settings TestMaker Enterprise 6.0.1 Licensed for 50 virtual users Report Name: Weekly Scalability Report and 10 service monitors Header: Weekly Scalability Report Footer: M Cohen - mcohen@bestbuy.com x3817 New Chart Edit Chart Delete Chart Charts Transaction Distribution Chart ! Scalability Index ! Transactions by Avg Duration Report Details Cancel Save
- 19. 19 Open Source Test Automation Intellectual Property of PushToTest. (c) 2011 All rights reserved. Chart Types ‣ Scalability Chart ‣ Bar Chart ‣ Grouped Data Points ‣ Transaction Chart ‣ Line and Data Point Chart ‣ Shows Individual Points of Data ‣ Resource Chart ‣ Bar Chart ‣ Resource (CPU, Net, Memory) Usage Percentage ‣ Data Chart ‣ Grid Chart ‣ Transaction and Step Data
- 20. 20 Open Source Test Automation Intellectual Property of PushToTest. (c) 2011 All rights reserved. Scalability Chart Settings ‣ Vertical ‣ Transactions Per Second ‣ Count of All Transactions ‣ Count of Failed Transactions ‣ Count of Successful Transactions ‣ Avg, Min, Max Transaction Time ‣ Horizontal: VU Iteration ‣ Secondary Axis (%) ‣ Failure Percentage ‣ Success Percentage ‣ Resource Monitoring (Choose from CPU, Net, Memory) ‣ Option: Display List of Step Names
- 21. 21 Open Source Test Automation Intellectual Property of PushToTest. (c) 2011 All rights reserved. Scalability Chart Settings PushToTest TestMaker 6 - Best Buy help TestMaker Results Repository Scalability Chart Settings TestMaker Enterprise 6.0.1 Licensed for 50 virtual users Transactions Per Second (TPS) and 10 service monitors Count of All Transactions Count of Successful Transactions Count of Failed Transactions Avg, Min, Max Transaction Time Average Secondary Axis Failure Percentage Success Percentage Resource Monitoring CPU Option: Display List of Step Names Cancel Save
- 22. 22 Open Source Test Automation Intellectual Property of PushToTest. (c) 2011 All rights reserved. Transaction Chart Settings ‣ Vertical ‣ Transaction Duration Time ‣ Vertical Options ‣ Successful ‣ Failed ‣ Duration: Avg, Min, Max ‣ Horizontal ‣ Time Offset Into Test ‣ Specific Start/End Time
- 23. 23 Open Source Test Automation Intellectual Property of PushToTest. (c) 2011 All rights reserved. Transaction Chart Settings PushToTest TestMaker 6 - Best Buy help TestMaker Results Repository Transaction Chart Settings TestMaker Enterprise 6.0.1 Licensed for 50 virtual users Transaction Duration Time and 10 service monitors Successful Failed Duration Average Horizontal Axis Show standard date/time Cancel Save
- 24. 24 Open Source Test Automation Intellectual Property of PushToTest. (c) 2011 All rights reserved. Resource Chart Settings ‣ Vertical ‣ Percentage used ‣ Grouped by CPU, Net, Memory ‣ Horizontal ‣ Time Offset Into Test ‣ Specific Start/End Time
- 25. 25 Open Source Test Automation Intellectual Property of PushToTest. (c) 2011 All rights reserved. Resource Chart Settings PushToTest TestMaker 6 - Best Buy help TestMaker Results Repository Resource Chart Settings TestMaker Enterprise 6.0.1 Licensed for 50 virtual users Percentage Used and 10 service monitors Grouped by CPU Horizontal Axis Show standard date/time Cancel Save
- 26. 26 Open Source Test Automation Intellectual Property of PushToTest. (c) 2011 All rights reserved. Data Chart Settings ‣ Successful Transactions ‣ Failed Transactions ‣ Option: Show Steps
- 27. 27 Open Source Test Automation Intellectual Property of PushToTest. (c) 2011 All rights reserved. Data Chart PushToTest TestMaker 6 - Best Buy help TestMaker Results Repository Data Chart Settings TestMaker Enterprise 6.0.1 Licensed for 50 virtual users and 10 service monitors Successful Transactions Failed Transactions Show Steps Cancel Save
- 28. 28 Open Source Test Automation Intellectual Property of PushToTest. (c) 2011 All rights reserved. Live Charts for Drill Downs PushToTest TestMaker 6 - Best Buy help TestMaker Results Repository Live Charts TestMaker Enterprise 6.0.1 Licensed for 50 virtual users and 10 service monitors New Chart View Chart Edit Chart Delete Chart Charts Transaction Distribution Chart Scalability Index Transactions by Avg Duration
- 29. 29 Open Source Test Automation Intellectual Property of PushToTest. (c) 2011 All rights reserved. Live Charts for Drill Downs PushToTest TestMaker 6 - Best Buy help TestMaker Results Repository TestMaker Enterprise 6.0.1 Licensed for 50 virtual users and 10 service monitors
- 30. 30 Open Source Test Automation Intellectual Property of PushToTest. (c) 2011 All rights reserved. Live Charts for Drill Downs PushToTest TestMaker 6 - Best Buy help TestMaker Results Repository TestMaker Enterprise 6.0.1 Licensed for 50 virtual users and 10 service monitors
- 31. 31 Open Source Test Automation Intellectual Property of PushToTest. (c) 2011 All rights reserved. Live Charts for Drill Downs PushToTest TestMaker 6 - Best Buy help TestMaker Results Repository TestMaker Enterprise 6.0.1 Licensed for 50 virtual users and 10 service monitors
- 32. 32 Open Source Test Automation Intellectual Property of PushToTest. (c) 2011 All rights reserved. Click To See Steps PushToTest TestMaker 6 - Best Buy help TestMaker Results Repository TestMaker Enterprise 6.0.1 Licensed for 50 virtual users and 10 service monitors Steps In Selected Transaction - soapUI TestSuite Request 1 175 Request 2 3,201 Groovy Script - Validate 643
- 33. 33 Open Source Test Automation Intellectual Property of PushToTest. (c) 2011 All rights reserved. Additional Requirements ‣ Keyboard left-right arrows move selected point ‣ User entered Notes ‣ Steps need labeling
- 34. 34 Open Source Test Automation Intellectual Property of PushToTest. (c) 2011 All rights reserved. Click Step To See Detail Steps In Selected Transaction - soapUI TestSuite Request 1 175 Request 2 3,201 Groovy Script - Validate 643 Request Response Error Custom <soapenv:Envelope xmlns:soapenv="http:// <soapenv:Header/> <soapenv:Body> <exam:Change_Price> <ProductNumber>1033</ProductNumber> <Type>add_product</Type> <Quantity>55</Quantity> <Location>2</Location>
- 35. 35 Open Source Test Automation Intellectual Property of PushToTest. (c) 2011 All rights reserved. For Java, DS Steps Steps In Selected Transaction - DesignerScript Log-in To Application 175 Product Catalog Search 3,201 Buy Products 675 Request Response Error Screenshot Command: _click Locator: textbox("company") Value: "PushToTest"
- 36. 36 Open Source Test Automation Intellectual Property of PushToTest. (c) 2011 All rights reserved. For Java, DS Steps Request Response Error Screenshot Row1_ColumnB: 38178 Row2_ColumnB: 98123 Row3_ColumnB: 18177 Request Response Error Screenshot Unable To Find Flight Number Exception java.lang.Throwable at boo.hoo.StackTrace$FirstNested$SecondNested. at boo.hoo.StackTrace$FirstNested.<init>(St at boo.hoo.StackTrace.<init>(StackTrace.j
- 37. 37 Open Source Test Automation Intellectual Property of PushToTest. (c) 2011 All rights reserved. Add A Step As A Chart Steps In Selected Transaction - DesignerScript Log-in To Application 175 Product Catalog Search 3,201 Buy Products 675 Add This Transaction To Report As New Chart Add This Step To Report As New Chart
- 38. 38 Open Source Test Automation Intellectual Property of PushToTest. (c) 2011 All rights reserved. Add More To The Report PushToTest TestMaker 6 - Weekly Scalability Report - Created on September 10, 2011 Weekly Scalability Report Created on September 10, 2011 Transaction Distribution Chart Step Detail Chart - Transaction 18372 - Step Log-in To Application Request Command: _click Locator: textbox("company") Value: "PushToTest" Response Row1_ColumnB: 38178 Row2_ColumnB: 98123 Row3_ColumnB: 18177 Error Unable To Find Flight Number Exception java.lang.Throwable at boo.hoo.StackTrace$FirstNested$SecondNested. at boo.hoo.StackTrace$FirstNested.<init>(St at boo.hoo.StackTrace.<init>(StackTrace.j Screenshot
- 39. Open Source Test Automation Intellectual Property of PushToTest. (c) 2011 All rights reserved. Next let’s talk about Resource Repository Design ‣ Resource Plan ‣ User Interface Design ‣ Required Changes to TestMaker 6.0 39
- 40. 40 Open Source Test Automation Intellectual Property of PushToTest. (c) 2011 All rights reserved. Repository Client Access ‣ Designer �� Editor ‣ Controller ‣ EOPs
- 41. 41 Open Source Test Automation Intellectual Property of PushToTest. (c) 2011 All rights reserved. Resource Repository - Open Open Test Resource Repositories Local Repository QA Unit 1 Repository / Last Modified Creator Project X June 12, 2011 6:05 pm mcohen 3rd Party Connector Project June 12, 2011 6:12 pm mcohen TestScenarios June 12, 2011 12:05 pm mcohen Load Test 103 Scenario June 12, 2011 1:05 pm mcohen Project Y June 12, 2011 3:05 pm mcohen Project Z June 12, 2011 5:05 pm mcohen Show Versions Cancel Open
- 42. 42 Open Source Test Automation Intellectual Property of PushToTest. (c) 2011 All rights reserved. Resource Repository - Save Save Test Resource Repositories Local Repository QA Unit 1 Repository / Last Modified Creator Project X June 12, 2011 6:05 pm mcohen 3rd Party Connector Project June 12, 2011 6:12 pm mcohen TestScenarios June 12, 2011 12:05 pm mcohen Load Test 103 Scenario June 12, 2011 1:05 pm mcohen Project Y June 12, 2011 3:05 pm mcohen Project Z June 12, 2011 5:05 pm mcohen Load Test 133 Scenario Show Versions Cancel Save
- 43. 43 Open Source Test Automation Intellectual Property of PushToTest. (c) 2011 All rights reserved. Resource Repository Preferences Repositories TestCard Server Notifications Repositories Local Repository Local Repository Type MySQL QA Unit 1 Repository Location jdbc:mysql//localhost:1527/testmaker User testmaker Add Remove Password testmaker Save
- 44. 44 Open Source Test Automation Intellectual Property of PushToTest. (c) 2011 All rights reserved. Learn More ‣ http://www.pushtotest.com ‣ Post your feedback, comments, ideas, designs to PushToTest Forums ‣ Attend the Open Source Test Workshop ‣ http://workshop.pushtotest.com ‣ Send Feedback to fcohen@pushtotest.com