SlideShare a Scribd company logo

TRASYS Testing As A Service

•
•
H
Hans Cortoos

TAAS is a model of software testing whereby TRASYS undertakes the activity of testing applications & solutions for our customers as a service with following traits: On-demand services service delivery function & service governance Well-defined & repeatable services (service catalogue, RACI, entry & exit criteria, ) Outsouring using a shared service centre Pay per use (based on test effort estimation techniques) Well defined test process & methodology Professional / certified test resources Supported by (cloud based and virtualised) test tools & test environments

Related slideshows

The Secrets to Real World Business Agility
The Secrets to Real World Business AgilityThe Secrets to Real World Business Agility
The Secrets to Real World Business Agility
 
What Is Agile Management?
What Is Agile Management?What Is Agile Management?
What Is Agile Management?
 
QA Challenge Accepted 4.0 - Cypress vs. Selenium
QA Challenge Accepted 4.0 - Cypress vs. SeleniumQA Challenge Accepted 4.0 - Cypress vs. Selenium
QA Challenge Accepted 4.0 - Cypress vs. Selenium
 
1 of 28
Download to read offline
© TRASYS 2008 TRASYS TAAS “Testing As A Service” November 2013
Š TRASYS 2008 Introduction
Testing As A Service: Definition TAAS is a model of software testing whereby TRASYS undertakes the activity of testing applications & solutions for our customers as a service with following traits:  On-demand services  service delivery function & service governance  Well-defined & repeatable services (service catalogue, RACI, entry & exit criteria, )  Outsouring using a shared service centre  Pay per use (based on test effort estimation techniques)  Well defined test process & methodology  Professional / certified test resources  Supported by (cloud based and virtualised) test tools & test environments
Testing as a Service: Key components Managed Services With KPIs and SLAs Centralised Resource allocation & global X-shore capability Defined list of services and test activities Remote, global and secure Acces to AUT and test environments Test tools as a service (SAAS Cloud) Certified and professional testers Unit price for each service (test case, use case , function point, LOC, etc) Test automation and simulators enhance productivity On demand and virtualised test environments Global Network of Test Centres Of Excellence Testing as a Service
Testing As A Service: Key Benefits Testing As a Service Benefits Reduce TCO - cost of quality (between 25% and 35%) Optimize governance & streamline communication Access to ‘niche’ expertise Predict service cost (fixed price) Shift from CAPEX to OPEX (pay for what you use) Schedule acceleration & flexibility (Rapid provisioning – shared HR and IT resources) Predict service outcome (result commitment – SLA’s – defined services) Reduce risk Levels & improve product & service quality
Š TRASYS 2008 Testing As A Service Service Delivery Model and assets
Testing as a Service: Our assets  ISO 9001:2008 certified Test Centre of Excellence  8 Western European locations and Ukraine  ISTQB, Prince2 and Agile certified testers  80% certified  Resource pool > 100  Majority of testers are senior or expert level ( between 5 and 10 years experience)  Test tools & test environments offered as a service  Secure remote access  Cloud based  Virtualized and on demand  See annex 1 for more details
Testing as a Service: Service Delivery Model Resource allocation Mgr. Release Test Centre Of Excellence SAAS test mgt. & defect mgt. tools Virtualised On Demand test environments Service catalogue Test Mangement & goveranance Nearshore team TRASYS Service Delivery Manager Testing best practices & metholodogy Client Service Manager Fixed Price Service Contract SLAs and metrics A Projects AApplications Simulators & test data mgt. tools On Demand Request Service Delivery Offshore team Releases On site team
Testing as a Service: Our vision • Business & technology risks define test strategy • Traceability matrix • Requirements coverage analysis Risk & Requirements Based • Test management & defect management • Test automation, test data analysis & mgt. • Performance testing & profiling • Security , penetration & vulnerability • Code review and inspection Tool supported • Focus on (important) business processes • Black box testers need to master the business • TCO & ROI drive test strategy & master plan Business driven • Based on best practices: ISTQB , ISO 9126, ISO 29119 • Test design techniques for test cases & test data • Test effort estimation techniques • (Master) test plan & formal test strategy Structured Test strategy, tools & checklists (= framework) for: • Mobile applications • SAP, CRM & ERP • Cloud SAAS • Webservices & SOA • Mainframe <> J2EE <> RIA & HTML5 <> … Technology specific • Test activities should be embedded in the SDLC • Testing starts at inception of a project / release • Testing should be adapted to the lifecycle: Agile, iterative, V-model Early & Adapted to lifecycle
Š TRASYS 2008 Testing As A Service Service Catalgue
Testing as a Service: Service Catalogue Data driven I.M test automation Code review & requirements review Cloud testing Keyword driven test automation Portability testing Mobile application testing Agile test driven Security & reliability Big data - DWH and BI testing Model based test automation Performance testing & profiling CRM and ERP Test automation Non functional testing Functional & business oriented Test solutions Integration and SOA testing E2E business process testing Usability & Crowd testing Functional system testing
Š TRASYS 2008 Testing As A Service Test Solutions
Testing as a Service: Test solutions Data driven I.M test automation Code review & requirements review Keyword driven test automation Portability testing Cloud testing Agile test driven Security & reliability Mobile application testing Model based test automation Performance testing & profiling Test automation Non functional testing CRM and ERP Functional & business oriented Test solutions Integration and SOA testing E2E business process testing Usability & Crowd testing Functional system testing Big data - DWH and BI testing
Testing as a Service: CRM & ERP Testing of a ‘commercial off-the-shelf’ database software application(s) supporting customer relationship management and enterprise resource planning business processes. Some leading providers of CRM / ERP Definition • Succesfull selection and Integration of the ‘package’ leading to ‘acceptance’ • (Test) Data accuracy, data quality and security during storage and retrieval • Understand the (impacted) business process(es) • Understand the test object ( platform) , the changes and their impact Key challenges • SAP solution manager, SAP TAO, SAP ECATT, SAP TDMS • Predefined test (automation) libraries & accelerators for SAP, Peoplesoft & Oracle (order to cash, procude to pay, etc) • HP QTP TRASYS tools • (automated) Data migration & platform migration testing • Customisation, parameterisation and usability testing • Assistance during evaluation / testing of prototypes - packages • (automated) version upgrade (regression) testing • Installability, configuration & deployment testing • E2E business process testing & system integration testing • (automated & data driven) functional acceptance testing and testing of reports • Application Lifecycle Test Management services ( ALM) TRASYS Services
Testing as a Service: Cloud Testing Testing conducted on the applications and services provided via cloud computing: ‘test object’ i.e application under test and / or testing services themselves are provided via cloud computing using PAAS / SAAS and APIs Defintion • Performance, scalability, capacity and availability • Access, privacy & security • Migration, interoperability & reversibility (avoid “vendor lock in”) Key challenges • HP Loadrunner • SOASTA Cloud test • SoapUI TRASYS tools • Load balance and stress testing for a range of simulated usage conditions • Role based & tool based security testing • Cloud migration testing (data migration, repository migration) • E2E system integration testing ( Hybrid cloud, interfaces between Cloud SAAS and back end systems and databases, API testing, SOA testing) TRASYS Services
Testing as a Service: Mobile Application Testing The functional and non functional testing of mobile-based native (or hybrid) applications. Some leading mobile app. platform providers: SAP, IBM, Kony, Antenna, Jquery mobile, Adobe, Appcelrator Definition • Understand the native application platform (hardware, OS) and network types • Testing needs to ascertain that the app can be successfully downloaded to the device, executed on the device and interact with the supporting backend & infrastructure and updates can be made (push out) • Understand the non functional requirements, risks & ‘vulnerabilities’ (security, performance, usability, portability…) related to mobile application development and (end user) usage Key challenges • Predefined mobile application checklists and ‘standard’ test cases (MAT starters kit) • Our mobile application test lab contains multiple fysical devices & emulators in order to support different test objectives. This is also supported by remote cloud based devices . • We can test multiple networks from our different locations (i.e GPRS tunneling effect) • Automated mobile test execution using emulators and test automation tools (see annex 1) TRASYS tools & assets • Risk based multidevice and multi OS testing, including backward compatibility • App lifecycle mgt testing ( from store readyness to push out of upgrade app versions) • Functional testing , security testing, performance testing, E2E business process testing, back end integration testing and usability testing for mobile applications TRASYS Services
Testing as a Service: Big Data, DWH and BI Testing of the data warehouse i.e the main organisations ’repository of transformed and cleaned historical data’ and business intelligence and decision support applications Some leading providers: Defintion • Availability of high quality (test) data in order to cover requirements (i.e reports or queries and ETLs) • High volatility of business rules & requirements in DWH/BI testing • Sensitivity, complexity & confidentiality of (test) data Key challenges • Tricentis Tosca Testsuite (for DWH testing) • Analytical simulation engine (accelerator) • Data driven testing tools and model based testing tools (See next slide) TRASYS tools • DWH, DI and Big data testing: • Source data extraction testing • ETL process validation (including statistical & dimensional analysis and data quality validation) • E2E integration testing of ETL batch jobs • Report validation, Cube validation, dashboard validation • Non functional testing (performance , security) • (Test) data mining, combined with ‘synthetic’ test data creation • Simulation of business rules while processing historical and/or near-to real time transactions in order to test systems TRASYS Services
Š TRASYS 2008 Testing As A Service Focus on specific services
Supporting test services Testing as a Service: Supporting test services Data driven I.M test automation Code review & requirements review Keyword driven test automation Portability testing Cloud testing Agile test driven Security & reliability Mobile application testing Model based test automation Performance testing & profiling Test automation Non functional testing CRM and ERP Functional & business oriented Test solutions Integration and SOA testing E2E business process testing Usability & Crowd testing Functional system testing Big data - DWH and BI testing
Testing as a Service: Model Based Testing (MBT) & Data Driven Testing (DDT) Testing based on a model of the component or system under test, e.g., reliability growth models, usage models such as operational profiles or behavioral models such as decision table or state transition diagram.Test cases are derived from a model such as finite state machine , UML, state chart, Markov chain that describes (mostly functional) aspects of the application under test -> Model based testing is supported by data driven test automation (DDT) -> Model based testing is a way of simulation early in the SDLC Defintion • Cost reduction • Increased test coverage • early validation of design model Key advantages • Test Optimal (MBT tool) • MaTeLo (MBT tool) • HP QTP (DDT automation tool) • Selenium ( test automation tool) TRASYS tools • Test data & test case generation using design techniques (boundary values, pair wise, branch, decision, transition, state, requirements,..) • Model based & data driven test automation • Test data management TRASYS Services
Testing as a Service: Security Testing Defintion Key challenges TRASYS tools • Vulnerability scanning & assessment • Penetration testing / attack based testing (= systematic probing of systems with the goal of gaining access to the information they contain) -> Open Web Application Security Project (OWASP) top 10 most critical web application flaws -> Open Source Security Testing Methodology (OSSTM) TRASYS Services • Acunetix • Webscarab • IBM Appscan • See appendix 1 for full list Determine that an information system protects data and prevents unauthorized access, whether accidental or deliberate, to programs and data. The six basic security concepts that need to be covered by security testing are: • confidentiality • Integrity • identification & authentication • Availability • Authorization • non-repudiation
Testing as a Service: Usability & Crowd Testing Defintion Key challenges TRASYS tools • Requirements based usability testing, including prototype & mock up validations during inception of SDLC using for example ISO 9241 • Real-world usage validation in order to continuously improve the systems * Use of production data analytics & usability tools * Crowd Testing = crowd sourcing + cloud test platform TRASYS Services • Chalkmark, Clickheat • Google analytics • Usertesting.com, Userzoom • Browser cam (cloud based) • UserReplay • Avoid that end users / customers leave (abandon) when a system is difficult to use, a homepage fails to clearly deliver information i.e E-commerce: if users cannot find the product, they cannot buy it either ( impact on revenue) • The functionality of our application is state of the art, but users lose time trying to figure out how to operate it ( impact on productivity) Testing the ease with which the user interfaces can be used and whether the application or the product built is user-friendly or not. Usability testing also reveals whether users feel comfortable with your application or Web site according to different parameters - the flow, navigation and layout, speed and content - especially in comparison to prior or similar applications.
Testing as a Service: Performance Testing Defintion Key challenges TRASYS tools • Load testing • Stress testing • Scalability testing & capacity planning support • Code review with performance focus – performance profiling • Reliability testing TRASYS Services • Loadrunner • Jmeter • Jprofiler • Several code review & inspection tools ( see Annex 1) • Avoid that end users / customers leave (abandon) when a system is ‘too slow’ to use, a homepage fails to deliver information with an acceptable response time -> impact on revenue • The functionality of our application is state of the art, but (internal or external) users lose time to operate and use the functionality -> impact on productivity • System is not available due to overload -> impact on revenue • System is oversized or undersized due to lack of performance testing -> impact on Capex / Opex The process of testing to determine the performance of a software product.
Š TRASYS 2008 Testing As A Service Implementation approach
Testing as a Service: Implementation approach Critical Success Factors Defined as major business transformation & major change IN YOUR ORGANISATION Managed as a project with a plan, budget , resources, scope, dependencies, risks,… Business case supported by share - and stakeholders Service contract with RACI and defined service inputs and outputs Collaboration model and governance model
Testing as a Service: Implementation approach Prepare phase •Feasibility study – Proof of Concept – Business Case •Test maturity assessment •Information and communication (generate awareness) Service Design phase •Target operating model & test process – governance model – collaboration model - RACI •Service contract – document of understanding (scope of services & projects / assets – pricing – planning) •Transition , change and deployment plan (including training and internal communication) Implementation phase •Knowledge transfer – information transfer – access to test object , test environments & tools •Proof of concepts •Mass deployment Operations phase •Run operations as a service •Monitor and report on operations (KPIs, activity reporting) •Manage demands and service delivery Improvement phase •Operational excellence through continuous test process improvement and service improvement
Annex 1 : Test tools Defect management Test Management Test automation Performance testing & profiling Code review & inspection Usability and prototype testing Security testing Agile testing Mobile application & cloud testing SAP testing HP Quality Centre - ALM HP Quality Centre - ALM HP QTP HP Loadrunner FindBugs Chalkmark Acunetix WVS Jira Greenhopper Fonemonkey SAP TAO Jira Spiratest Selenium Jmeter PMD Clickheat BackTrack SmartBear ALM Perfectomobile (cloud based) SAP Solution Manager Bugzilla Flexmonkey Jprofiler CheckStyle Google analytics Paros Proxy MS Visual Studio and TFS IBM Worklight (mobile) and IBM Smartcloud SAP eCATT Mantis SOASTA JavaNCSS Usertesting. com Tamper Data Mylyn for eclipse SOASTA (cloud based) • CloudTest • TouchTest HP QTP SAP plug in SOAP UI (SOA testing) Cobertura Userzoom WebScarab HP Agile Manager (cloud based) Open Cloud architecture ( Cloud Foundry) Test Optimal (model based testing) SonarQube Google Website Optimizer CSRF Tester W3C mobile ok checker MaTeLo (model based testing) Browser cam (cloud based) IBM rational AppScan Deviceanywhere (cloud based) UserReplay Xamarin (cloud based)
Thank you for your attention! 28TRASYS Confidential

More Related Content

TRASYS Testing As A Service

  • 1. Š TRASYS 2008 TRASYS TAAS “Testing As A Service” November 2013
  • 3. Testing As A Service: Definition TAAS is a model of software testing whereby TRASYS undertakes the activity of testing applications & solutions for our customers as a service with following traits:  On-demand services  service delivery function & service governance  Well-defined & repeatable services (service catalogue, RACI, entry & exit criteria, )  Outsouring using a shared service centre  Pay per use (based on test effort estimation techniques)  Well defined test process & methodology  Professional / certified test resources  Supported by (cloud based and virtualised) test tools & test environments
  • 4. Testing as a Service: Key components Managed Services With KPIs and SLAs Centralised Resource allocation & global X-shore capability Defined list of services and test activities Remote, global and secure Acces to AUT and test environments Test tools as a service (SAAS Cloud) Certified and professional testers Unit price for each service (test case, use case , function point, LOC, etc) Test automation and simulators enhance productivity On demand and virtualised test environments Global Network of Test Centres Of Excellence Testing as a Service
  • 5. Testing As A Service: Key Benefits Testing As a Service Benefits Reduce TCO - cost of quality (between 25% and 35%) Optimize governance & streamline communication Access to ‘niche’ expertise Predict service cost (fixed price) Shift from CAPEX to OPEX (pay for what you use) Schedule acceleration & flexibility (Rapid provisioning – shared HR and IT resources) Predict service outcome (result commitment – SLA’s – defined services) Reduce risk Levels & improve product & service quality
  • 6. Š TRASYS 2008 Testing As A Service Service Delivery Model and assets
  • 7. Testing as a Service: Our assets  ISO 9001:2008 certified Test Centre of Excellence  8 Western European locations and Ukraine  ISTQB, Prince2 and Agile certified testers  80% certified  Resource pool > 100  Majority of testers are senior or expert level ( between 5 and 10 years experience)  Test tools & test environments offered as a service  Secure remote access  Cloud based  Virtualized and on demand  See annex 1 for more details
  • 8. Testing as a Service: Service Delivery Model Resource allocation Mgr. Release Test Centre Of Excellence SAAS test mgt. & defect mgt. tools Virtualised On Demand test environments Service catalogue Test Mangement & goveranance Nearshore team TRASYS Service Delivery Manager Testing best practices & metholodogy Client Service Manager Fixed Price Service Contract SLAs and metrics A Projects AApplications Simulators & test data mgt. tools On Demand Request Service Delivery Offshore team Releases On site team
  • 9. Testing as a Service: Our vision • Business & technology risks define test strategy • Traceability matrix • Requirements coverage analysis Risk & Requirements Based • Test management & defect management • Test automation, test data analysis & mgt. • Performance testing & profiling • Security , penetration & vulnerability • Code review and inspection Tool supported • Focus on (important) business processes • Black box testers need to master the business • TCO & ROI drive test strategy & master plan Business driven • Based on best practices: ISTQB , ISO 9126, ISO 29119 • Test design techniques for test cases & test data • Test effort estimation techniques • (Master) test plan & formal test strategy Structured Test strategy, tools & checklists (= framework) for: • Mobile applications • SAP, CRM & ERP • Cloud SAAS • Webservices & SOA • Mainframe <> J2EE <> RIA & HTML5 <> … Technology specific • Test activities should be embedded in the SDLC • Testing starts at inception of a project / release • Testing should be adapted to the lifecycle: Agile, iterative, V-model Early & Adapted to lifecycle
  • 10. Š TRASYS 2008 Testing As A Service Service Catalgue
  • 11. Testing as a Service: Service Catalogue Data driven I.M test automation Code review & requirements review Cloud testing Keyword driven test automation Portability testing Mobile application testing Agile test driven Security & reliability Big data - DWH and BI testing Model based test automation Performance testing & profiling CRM and ERP Test automation Non functional testing Functional & business oriented Test solutions Integration and SOA testing E2E business process testing Usability & Crowd testing Functional system testing
  • 12. Š TRASYS 2008 Testing As A Service Test Solutions
  • 13. Testing as a Service: Test solutions Data driven I.M test automation Code review & requirements review Keyword driven test automation Portability testing Cloud testing Agile test driven Security & reliability Mobile application testing Model based test automation Performance testing & profiling Test automation Non functional testing CRM and ERP Functional & business oriented Test solutions Integration and SOA testing E2E business process testing Usability & Crowd testing Functional system testing Big data - DWH and BI testing
  • 14. Testing as a Service: CRM & ERP Testing of a ‘commercial off-the-shelf’ database software application(s) supporting customer relationship management and enterprise resource planning business processes. Some leading providers of CRM / ERP Definition • Succesfull selection and Integration of the ‘package’ leading to ‘acceptance’ • (Test) Data accuracy, data quality and security during storage and retrieval • Understand the (impacted) business process(es) • Understand the test object ( platform) , the changes and their impact Key challenges • SAP solution manager, SAP TAO, SAP ECATT, SAP TDMS • Predefined test (automation) libraries & accelerators for SAP, Peoplesoft & Oracle (order to cash, procude to pay, etc) • HP QTP TRASYS tools • (automated) Data migration & platform migration testing • Customisation, parameterisation and usability testing • Assistance during evaluation / testing of prototypes - packages • (automated) version upgrade (regression) testing • Installability, configuration & deployment testing • E2E business process testing & system integration testing • (automated & data driven) functional acceptance testing and testing of reports • Application Lifecycle Test Management services ( ALM) TRASYS Services
  • 15. Testing as a Service: Cloud Testing Testing conducted on the applications and services provided via cloud computing: ‘test object’ i.e application under test and / or testing services themselves are provided via cloud computing using PAAS / SAAS and APIs Defintion • Performance, scalability, capacity and availability • Access, privacy & security • Migration, interoperability & reversibility (avoid “vendor lock in”) Key challenges • HP Loadrunner • SOASTA Cloud test • SoapUI TRASYS tools • Load balance and stress testing for a range of simulated usage conditions • Role based & tool based security testing • Cloud migration testing (data migration, repository migration) • E2E system integration testing ( Hybrid cloud, interfaces between Cloud SAAS and back end systems and databases, API testing, SOA testing) TRASYS Services
  • 16. Testing as a Service: Mobile Application Testing The functional and non functional testing of mobile-based native (or hybrid) applications. Some leading mobile app. platform providers: SAP, IBM, Kony, Antenna, Jquery mobile, Adobe, Appcelrator Definition • Understand the native application platform (hardware, OS) and network types • Testing needs to ascertain that the app can be successfully downloaded to the device, executed on the device and interact with the supporting backend & infrastructure and updates can be made (push out) • Understand the non functional requirements, risks & ‘vulnerabilities’ (security, performance, usability, portability…) related to mobile application development and (end user) usage Key challenges • Predefined mobile application checklists and ‘standard’ test cases (MAT starters kit) • Our mobile application test lab contains multiple fysical devices & emulators in order to support different test objectives. This is also supported by remote cloud based devices . • We can test multiple networks from our different locations (i.e GPRS tunneling effect) • Automated mobile test execution using emulators and test automation tools (see annex 1) TRASYS tools & assets • Risk based multidevice and multi OS testing, including backward compatibility • App lifecycle mgt testing ( from store readyness to push out of upgrade app versions) • Functional testing , security testing, performance testing, E2E business process testing, back end integration testing and usability testing for mobile applications TRASYS Services
  • 17. Testing as a Service: Big Data, DWH and BI Testing of the data warehouse i.e the main organisations ’repository of transformed and cleaned historical data’ and business intelligence and decision support applications Some leading providers: Defintion • Availability of high quality (test) data in order to cover requirements (i.e reports or queries and ETLs) • High volatility of business rules & requirements in DWH/BI testing • Sensitivity, complexity & confidentiality of (test) data Key challenges • Tricentis Tosca Testsuite (for DWH testing) • Analytical simulation engine (accelerator) • Data driven testing tools and model based testing tools (See next slide) TRASYS tools • DWH, DI and Big data testing: • Source data extraction testing • ETL process validation (including statistical & dimensional analysis and data quality validation) • E2E integration testing of ETL batch jobs • Report validation, Cube validation, dashboard validation • Non functional testing (performance , security) • (Test) data mining, combined with ‘synthetic’ test data creation • Simulation of business rules while processing historical and/or near-to real time transactions in order to test systems TRASYS Services
  • 18. Š TRASYS 2008 Testing As A Service Focus on specific services
  • 19. Supporting test services Testing as a Service: Supporting test services Data driven I.M test automation Code review & requirements review Keyword driven test automation Portability testing Cloud testing Agile test driven Security & reliability Mobile application testing Model based test automation Performance testing & profiling Test automation Non functional testing CRM and ERP Functional & business oriented Test solutions Integration and SOA testing E2E business process testing Usability & Crowd testing Functional system testing Big data - DWH and BI testing
  • 20. Testing as a Service: Model Based Testing (MBT) & Data Driven Testing (DDT) Testing based on a model of the component or system under test, e.g., reliability growth models, usage models such as operational profiles or behavioral models such as decision table or state transition diagram.Test cases are derived from a model such as finite state machine , UML, state chart, Markov chain that describes (mostly functional) aspects of the application under test -> Model based testing is supported by data driven test automation (DDT) -> Model based testing is a way of simulation early in the SDLC Defintion • Cost reduction • Increased test coverage • early validation of design model Key advantages • Test Optimal (MBT tool) • MaTeLo (MBT tool) • HP QTP (DDT automation tool) • Selenium ( test automation tool) TRASYS tools • Test data & test case generation using design techniques (boundary values, pair wise, branch, decision, transition, state, requirements,..) • Model based & data driven test automation • Test data management TRASYS Services
  • 21. Testing as a Service: Security Testing Defintion Key challenges TRASYS tools • Vulnerability scanning & assessment • Penetration testing / attack based testing (= systematic probing of systems with the goal of gaining access to the information they contain) -> Open Web Application Security Project (OWASP) top 10 most critical web application flaws -> Open Source Security Testing Methodology (OSSTM) TRASYS Services • Acunetix • Webscarab • IBM Appscan • See appendix 1 for full list Determine that an information system protects data and prevents unauthorized access, whether accidental or deliberate, to programs and data. The six basic security concepts that need to be covered by security testing are: • confidentiality • Integrity • identification & authentication • Availability • Authorization • non-repudiation
  • 22. Testing as a Service: Usability & Crowd Testing Defintion Key challenges TRASYS tools • Requirements based usability testing, including prototype & mock up validations during inception of SDLC using for example ISO 9241 • Real-world usage validation in order to continuously improve the systems * Use of production data analytics & usability tools * Crowd Testing = crowd sourcing + cloud test platform TRASYS Services • Chalkmark, Clickheat • Google analytics • Usertesting.com, Userzoom • Browser cam (cloud based) • UserReplay • Avoid that end users / customers leave (abandon) when a system is difficult to use, a homepage fails to clearly deliver information i.e E-commerce: if users cannot find the product, they cannot buy it either ( impact on revenue) • The functionality of our application is state of the art, but users lose time trying to figure out how to operate it ( impact on productivity) Testing the ease with which the user interfaces can be used and whether the application or the product built is user-friendly or not. Usability testing also reveals whether users feel comfortable with your application or Web site according to different parameters - the flow, navigation and layout, speed and content - especially in comparison to prior or similar applications.
  • 23. Testing as a Service: Performance Testing Defintion Key challenges TRASYS tools • Load testing • Stress testing • Scalability testing & capacity planning support • Code review with performance focus – performance profiling • Reliability testing TRASYS Services • Loadrunner • Jmeter • Jprofiler • Several code review & inspection tools ( see Annex 1) • Avoid that end users / customers leave (abandon) when a system is ‘too slow’ to use, a homepage fails to deliver information with an acceptable response time -> impact on revenue • The functionality of our application is state of the art, but (internal or external) users lose time to operate and use the functionality -> impact on productivity • System is not available due to overload -> impact on revenue • System is oversized or undersized due to lack of performance testing -> impact on Capex / Opex The process of testing to determine the performance of a software product.
  • 24. Š TRASYS 2008 Testing As A Service Implementation approach
  • 25. Testing as a Service: Implementation approach Critical Success Factors Defined as major business transformation & major change IN YOUR ORGANISATION Managed as a project with a plan, budget , resources, scope, dependencies, risks,… Business case supported by share - and stakeholders Service contract with RACI and defined service inputs and outputs Collaboration model and governance model
  • 26. Testing as a Service: Implementation approach Prepare phase •Feasibility study – Proof of Concept – Business Case •Test maturity assessment •Information and communication (generate awareness) Service Design phase •Target operating model & test process – governance model – collaboration model - RACI •Service contract – document of understanding (scope of services & projects / assets – pricing – planning) •Transition , change and deployment plan (including training and internal communication) Implementation phase •Knowledge transfer – information transfer – access to test object , test environments & tools •Proof of concepts •Mass deployment Operations phase •Run operations as a service •Monitor and report on operations (KPIs, activity reporting) •Manage demands and service delivery Improvement phase •Operational excellence through continuous test process improvement and service improvement
  • 27. Annex 1 : Test tools Defect management Test Management Test automation Performance testing & profiling Code review & inspection Usability and prototype testing Security testing Agile testing Mobile application & cloud testing SAP testing HP Quality Centre - ALM HP Quality Centre - ALM HP QTP HP Loadrunner FindBugs Chalkmark Acunetix WVS Jira Greenhopper Fonemonkey SAP TAO Jira Spiratest Selenium Jmeter PMD Clickheat BackTrack SmartBear ALM Perfectomobile (cloud based) SAP Solution Manager Bugzilla Flexmonkey Jprofiler CheckStyle Google analytics Paros Proxy MS Visual Studio and TFS IBM Worklight (mobile) and IBM Smartcloud SAP eCATT Mantis SOASTA JavaNCSS Usertesting. com Tamper Data Mylyn for eclipse SOASTA (cloud based) • CloudTest • TouchTest HP QTP SAP plug in SOAP UI (SOA testing) Cobertura Userzoom WebScarab HP Agile Manager (cloud based) Open Cloud architecture ( Cloud Foundry) Test Optimal (model based testing) SonarQube Google Website Optimizer CSRF Tester W3C mobile ok checker MaTeLo (model based testing) Browser cam (cloud based) IBM rational AppScan Deviceanywhere (cloud based) UserReplay Xamarin (cloud based)
  • 28. Thank you for your attention! 28TRASYS Confidential