SlideShare a Scribd company logo
Non-Functional Testing
www.kanoah.com
ABOUT US
Kanoah is an innovative company
providing ground-breaking solutions to
software testing professionals on the
Atlassian JIRA platform
About Kanoah Tests
Kanoah Tests is a full featured test management, integrated seamless into
JIRA with the same look-n-feel. No need to learn or switch between different
applications
Coordinate all test management activities including test planning, authoring,
execution, tracking and reporting from a central location
Kanoah Tests enables you to track testing progress and quality to foster
collaboration and visibility across traditional and agile teams
Get real-time insights into your testing progress with out of the box
reports
Easily integrate your automated tests and submit test results with Kanoah
Tests’ powerful REST API or use the API to automate many areas of the
application
After looking for several years at plugins for test management we finally found
Kanoah Tests. The other solutions were either too complex, didn't integrate well
with Jira, or were focused on a single project. Kanoah Tests proved to be an
elegant solution that allowed linking between any project. Kanoah has been
very responsive to feedback, requests, suggestions as well bugs. The customer
service is awesome. I'd highly recommend Kanoah Tests to teams of any size
looking to simplify test management and consolidate tools.
Don
Pierce
Robert
Murhamer
Liked Kanoah from the moment I discovered it. Integrates nicely with JIRA and
especially with Agile. Test cases can be authored right from the story level, but has
all other functionality a Test Case Management solution would need to have,
ranging from creating test plans, executing test cases, importing test cases, API for
automation, a.s.o. Additionally the team at Kanoah is amazing and responding to
any question very quickly. It wasn't hard to sell to my management to purchase
Kanoah. Will highly recommend Kanoah to anybody.
Zour
Brosh
I just start working with Kanoah and I am impressed how it's simple to manage tests
without unlimited non-used features like in most of the test management tools and
still to get the needed functionality and results. The integration with Jira is a great
working solution that enable to share testing and development in simple way on
one system. I recommend to use Kanoah for testing management. It will help to do
Kanoah as much as possible customizable like Jira to match each group
methodology
For more reviews, visit: https://marketplace.atlassian.com/plugins/com.kanoah.test-manager/server/reviews
Reviews
Key Features
Native seamless integration with JIRA
No need to learn or switch between
different applications
Perfect for agile & traditional testing
approaches
Manage, organize and track all your testing
efforts in a central place
Reuse test cases across your projects
Powerful REST API
Establish clear traceability between
requirements, test cases, and defects
Execute test cases and track results that
matter
Get real-time insights into your testing
progress with out of the box reports
Live statistics accessible to your entire
team
Benefits for the testers
No need to lear or switch between
different applications
Reuse test cases across projects for
regression
Link test cases to requirements and
defects
API support for automated efforts
Benefits for the teams
Informed decisions based on real-time
insights
End-to-end traceability and impact
analysis
Centralized Test Management
Save time and increase productivity
Why choose Kanoah Tests
Coordinate all test
management activities right
inside JIRA
1Testing right
inside JIRA
Take advantage of the built-in
reports to track the results and
measure progress
2Make informed
decisions
Kanoah Tests users receive
priority support, even
during trials
3Stelar
support
Non-Functional Testing
What is Non-Functional Testing?
Non-functional Testing is the testing of a software application or system for its
non-functional requirements: the way a system operates, rather than specific
behaviours of that system. [1]
Non-functional Testing is concerned with the non-functional requirements and
is designed specifically to evaluate the readiness of a system according to the
various criteria which are not covered by functional testing. [2]
Non-functional requirements tend to be those that reflect the quality of the
product, particularly in the context of the suitability perspective of its users. It
can be started after the completion of Functional Testing. The non functional
tests can be effective by using testing tools. [3]
What is Non-Functional Testing?
Non-functional Testing implies the testing of software attributes which are not
related to any specific function or user action like performance, scalability,
security or behavior of application under certain constraints.
Non-functional testing has a great influence on customer and user
satisfaction with the product. Non functional testing should be expressed in a
testable way, not like “the system should be fast” or “the system should be
easy to operate” which is not testable. [3]
Types of Non-Functional Testing
The following types and definitions have been recompiled from these authors [1]
[2] [3].
Reliability testing: Reliability Testing is about exercising an application so that
failures are discovered and removed before the system is deployed. The
purpose is to determine product reliability, and to determine whether the
software meets the customer’s reliability requirements.
Efficiency testing: Efficiency testing test the amount of code and testing
resources required by a program to perform a particular function. Software
Test Efficiency is number of test cases executed divided by unit of time
(generally per hour).
Usability testing: The testers tests the ease with which the user interfaces can
be used. It tests that whether the application or the product built is user-
friendly or not. It also includes five components:
1. Learnability: How easy is it for users to accomplish basic tasks the first
time they encounter the design?
2. Efficiency: How fast can experienced users accomplish tasks?
3. Memorability: When users return to the design after a period of not using
it, does the user remember enough to use it effectively the next time, or
does the user have to start over again learning everything?
4. Errors: How many errors do users make, how severe are these errors and
how easily can they recover from the errors?
5. Satisfaction: How much does the user like using the system?
Types of Non-Functional Testing
Types of Non-Functional Testing
Maintainability testing: It basically defines that how easy it is to maintain the
system. This means that how easy it is to analyze, change and test the
application or product.
Portability testing: It refers to the process of testing the ease with which a
computer software component or application can be moved from one
environment to another, e.g. moving of any application from Windows 2000 to
Windows XP. This is usually measured in terms of the maximum amount of
effort permitted. Results are measured in terms of the time required to move
the software and complete the and documentation updates.
Baseline testing: It refers to the validation of documents and specifications on
which test cases would be designed. The requirement specification validation
is baseline testing.
Types of Non-Functional Testing
Compliance testing: It is related with the IT standards followed by the
company and it is the testing done to find the deviations from the company
prescribed standards.
Documentation testing: As per the IEEE Documentation describing plans for, or
results of, the testing of a system or component, Types include test case
specification, test incident report, test log, test plan, test procedure, test
report. Hence the testing of all the above mentioned documents is known as
documentation testing.
Endurance testing: Endurance testing involves testing a system with a
significant load extended over a significant period of time, to discover how the
system behaves under sustained use. For example, in software testing, a
system may behave exactly as expected when tested for 1 hour but when the
same system is tested for 3 hours, problems such as memory leaks cause the
system to fail or behave randomly.
Types of Non-Functional Testing
Load testing: Load testing is performed to determine a system’s behavior
under both normal and at peak conditions. It helps to identify the maximum
operating capacity of an application as well as any bottlenecks and determine
which element is causing degradation.
Activities of Load Testing might include:
Validate that the system performs as expected when concurrent users
access the application and gets the expected response time.
This test is repeated with multiple users to get the response time and
throughput.
At the time of testing, the data base should be realistic.
The test should be conducted on a dedicated server which stimulates the
actual environment.
Types of Non-Functional Testing
Performance testing: Performance testing is performed to determine how fast
some aspect of a system performs under a particular workload. It can serve
different purposes like it can demonstrate that the system meets performance
criteria. It can compare two systems to find which performs better. Or it can
measure what part of the system or workload causes the system to perform
badly.
Activities in Performance Testing might include:
Validate that the system meets the expected response time.
Evaluate that the significant elements of the application meets the desired
response time.
It can also be conducted as a pert of integration testing.
It can also be conducted as a part of systems testing.
Types of Non-Functional Testing
Compatibility testing: Compatibility testing is basically the testing of the
application or the product built with the computing environment. It tests
whether the application or the software product built is compatible with the
hardware, operating system, database or other system software or not.
Activities in Compatibility Testing might include:
Test with each hardware with minimum and maximum configuration.
Test with different browsers.
Test cases are the same which were executed during functional testing.
In case the number of hardware and software are too many, we can use
OATS techniques to arrive at the test cases to have maximum coverage.
Types of Non-Functional Testing
Security testing: Security testing is basically to check that whether the
application or the product is secured or not. Can anyone came tomorrow and
hack the system or login the application without any authorization? It is a
process to determine that an information system protects data and maintains
functionality as intended.
Scalability testing: It is the testing of a software application for measuring its
capability to scale up in terms of any of its non-functional capability like load
supported, the number of transactions, the data volume etc.
Types of Non-Functional Testing
Volume testing: Volume testing refers to testing a software application or the
product with a certain amount of data. E.g., if we want to volume test our
application with a specific database size, we need to expand our database to
that size and then test the application’s performance on it.
Some key points of Volume Testing:
Subject the software to large amounts of data and check the limit where the
software fails.
Maximum database size is created and multiple client query the database or
create larger report.
If the application is processing database to create a report, a volume test
would be to use a large result set and check that the report is printed
correctly.
Types of Non-Functional Testing
Stress Testing: It is a form of testing that is used to determine the stability of a
given system. It puts greater emphasis on robustness, availability, and error
handling under a heavy load, rather than on what would be considered
correct behavior under normal circumstances. The goals of such tests may be
to ensure the software does not crash in conditions of insufficient
computational resources (such as memory or disk space).
Activities of Stress Testing might include:
Test on low memory or low disc space on clients or servers that reveals the
defects which cannot be found under normal conditions.
Multiple user performance the same transactions on the same data.
Multiple clients connected to servers with different workloads.
Reduce the Think Time to “Zero” to stress the servers to their maximum
stress.
Types of Non-Functional Testing
Recovery testing: Recovery testing is done in order to check how fast and
better the application can recover after it has gone through any type of crash
or hardware failure (etc). Recovery testing is the forced failure of the software
in a variety of ways to verify that recovery is properly performed.
Recovery Testing activities might include:
Power interruption, to the client while doing CURD activities.
Invalid database pointers and keys
Database process are aborted or prematurely terminated.
Database pointers, fields and keys are corrupted manually and directly
within database.
Physically disconnect the communication wires, power turn off, turn down
the routers and network servers.
Types of Non-Functional Testing
Internationalization testing: Internationalization is a process of designing a
software application so that it can be adapted to various languages and
regions without any changes.
Localization Testing: Localization is a process of adapting internationalized
software for a specific region or language by adding local specific
components and translating text.
Estimating Non-Functional Requirements
Once the team accepts a non-functional requirement into the project they
need to remain in compliance with that non-functional requirement for the
remainder of the project.
Doing performance testing (or staying in compliance with any non-functional
requirement) creates some amount of overhead on the team. This overhead,
or tax, must be paid regularly. In some cases, the team and product owner will
decide the tax must be paid every sprint. In other cases the team and
product owner may agree to pay the tax every few sprints.
Estimate the cost of initial compliance just like any other user story or product
backlog item. The team and product owner will need to incorporate an
estimate of when they'll do this. Adding performance testing after five sprints
is different than adding it after 20 sprints. [5]
Distinction between Functional and Non Functional Testing
Functional testing ensures that your product meets customer and business
requirements, and doesn’t have any major bugs.
Functional testing is designed to determine that the application’s features and
operations perform the way they should.
Non-functional testing wants to see if the product stands up to customer
expectations.
Non-functional testing wants to know that the product “behaves” correctly.
Failure to release a working product that meets the needs of consumer demands
can damage your company’s reputation and reduce overall product sales. [6]
Functional Testing
The following list has been extracted from here [7].
Tests how well the system performs.
Based on client requirements.
Means testing the application against business requirements.
It is a part of System Testing.
Validates the behavior of the application.
Non-Functional Testing
The following list has been extracted from here [7].
Tests how well the system responds.
Based on client expectations.
Means testing the application against the client and performance
requirements.
Also a part of System Testing.
Validates the performance of the application.
References
[1] https://en.wikipedia.org/wiki/Non-functional_testing
[2] http://reqtest.com/testing-blog/functional-vs-non-functional-testing/
[3] http://www.softwaretestinghelp.com/what-is-non-functional-testing/
[4] https://www.mountaingoatsoftware.com/blog/non-functional-requirements-
as-user-stories
[5] https://www.mountaingoatsoftware.com/blog/estimating-non-functional-
requirements
[6] http://www.ibeta.com/functional-vs-non-functional-testing-whats-difference/
[7] http://testingbasicinterviewquestions.blogspot.com.br/2015/02/functional-
testing-vs-non-functional-testing.html
Next Steps
Getting Started Guide
GETTING STARTED RESOURCES FREE TRIAL
Documentation
Support
Tutorials
What we're reading this week
Blog
Start you 30-day free trial now

More Related Content

Non-Functional testing

  • 2. www.kanoah.com ABOUT US Kanoah is an innovative company providing ground-breaking solutions to software testing professionals on the Atlassian JIRA platform
  • 3. About Kanoah Tests Kanoah Tests is a full featured test management, integrated seamless into JIRA with the same look-n-feel. No need to learn or switch between different applications Coordinate all test management activities including test planning, authoring, execution, tracking and reporting from a central location Kanoah Tests enables you to track testing progress and quality to foster collaboration and visibility across traditional and agile teams Get real-time insights into your testing progress with out of the box reports Easily integrate your automated tests and submit test results with Kanoah Tests’ powerful REST API or use the API to automate many areas of the application
  • 4. After looking for several years at plugins for test management we finally found Kanoah Tests. The other solutions were either too complex, didn't integrate well with Jira, or were focused on a single project. Kanoah Tests proved to be an elegant solution that allowed linking between any project. Kanoah has been very responsive to feedback, requests, suggestions as well bugs. The customer service is awesome. I'd highly recommend Kanoah Tests to teams of any size looking to simplify test management and consolidate tools. Don Pierce Robert Murhamer Liked Kanoah from the moment I discovered it. Integrates nicely with JIRA and especially with Agile. Test cases can be authored right from the story level, but has all other functionality a Test Case Management solution would need to have, ranging from creating test plans, executing test cases, importing test cases, API for automation, a.s.o. Additionally the team at Kanoah is amazing and responding to any question very quickly. It wasn't hard to sell to my management to purchase Kanoah. Will highly recommend Kanoah to anybody. Zour Brosh I just start working with Kanoah and I am impressed how it's simple to manage tests without unlimited non-used features like in most of the test management tools and still to get the needed functionality and results. The integration with Jira is a great working solution that enable to share testing and development in simple way on one system. I recommend to use Kanoah for testing management. It will help to do Kanoah as much as possible customizable like Jira to match each group methodology For more reviews, visit: https://marketplace.atlassian.com/plugins/com.kanoah.test-manager/server/reviews Reviews
  • 5. Key Features Native seamless integration with JIRA No need to learn or switch between different applications Perfect for agile & traditional testing approaches Manage, organize and track all your testing efforts in a central place Reuse test cases across your projects Powerful REST API Establish clear traceability between requirements, test cases, and defects Execute test cases and track results that matter Get real-time insights into your testing progress with out of the box reports Live statistics accessible to your entire team
  • 6. Benefits for the testers No need to lear or switch between different applications Reuse test cases across projects for regression Link test cases to requirements and defects API support for automated efforts Benefits for the teams Informed decisions based on real-time insights End-to-end traceability and impact analysis Centralized Test Management Save time and increase productivity
  • 7. Why choose Kanoah Tests Coordinate all test management activities right inside JIRA 1Testing right inside JIRA Take advantage of the built-in reports to track the results and measure progress 2Make informed decisions Kanoah Tests users receive priority support, even during trials 3Stelar support
  • 9. What is Non-Functional Testing? Non-functional Testing is the testing of a software application or system for its non-functional requirements: the way a system operates, rather than specific behaviours of that system. [1] Non-functional Testing is concerned with the non-functional requirements and is designed specifically to evaluate the readiness of a system according to the various criteria which are not covered by functional testing. [2] Non-functional requirements tend to be those that reflect the quality of the product, particularly in the context of the suitability perspective of its users. It can be started after the completion of Functional Testing. The non functional tests can be effective by using testing tools. [3]
  • 10. What is Non-Functional Testing? Non-functional Testing implies the testing of software attributes which are not related to any specific function or user action like performance, scalability, security or behavior of application under certain constraints. Non-functional testing has a great influence on customer and user satisfaction with the product. Non functional testing should be expressed in a testable way, not like “the system should be fast” or “the system should be easy to operate” which is not testable. [3]
  • 11. Types of Non-Functional Testing The following types and definitions have been recompiled from these authors [1] [2] [3]. Reliability testing: Reliability Testing is about exercising an application so that failures are discovered and removed before the system is deployed. The purpose is to determine product reliability, and to determine whether the software meets the customer’s reliability requirements. Efficiency testing: Efficiency testing test the amount of code and testing resources required by a program to perform a particular function. Software Test Efficiency is number of test cases executed divided by unit of time (generally per hour).
  • 12. Usability testing: The testers tests the ease with which the user interfaces can be used. It tests that whether the application or the product built is user- friendly or not. It also includes five components: 1. Learnability: How easy is it for users to accomplish basic tasks the first time they encounter the design? 2. Efficiency: How fast can experienced users accomplish tasks? 3. Memorability: When users return to the design after a period of not using it, does the user remember enough to use it effectively the next time, or does the user have to start over again learning everything? 4. Errors: How many errors do users make, how severe are these errors and how easily can they recover from the errors? 5. Satisfaction: How much does the user like using the system? Types of Non-Functional Testing
  • 13. Types of Non-Functional Testing Maintainability testing: It basically defines that how easy it is to maintain the system. This means that how easy it is to analyze, change and test the application or product. Portability testing: It refers to the process of testing the ease with which a computer software component or application can be moved from one environment to another, e.g. moving of any application from Windows 2000 to Windows XP. This is usually measured in terms of the maximum amount of effort permitted. Results are measured in terms of the time required to move the software and complete the and documentation updates. Baseline testing: It refers to the validation of documents and specifications on which test cases would be designed. The requirement specification validation is baseline testing.
  • 14. Types of Non-Functional Testing Compliance testing: It is related with the IT standards followed by the company and it is the testing done to find the deviations from the company prescribed standards. Documentation testing: As per the IEEE Documentation describing plans for, or results of, the testing of a system or component, Types include test case specification, test incident report, test log, test plan, test procedure, test report. Hence the testing of all the above mentioned documents is known as documentation testing. Endurance testing: Endurance testing involves testing a system with a significant load extended over a significant period of time, to discover how the system behaves under sustained use. For example, in software testing, a system may behave exactly as expected when tested for 1 hour but when the same system is tested for 3 hours, problems such as memory leaks cause the system to fail or behave randomly.
  • 15. Types of Non-Functional Testing Load testing: Load testing is performed to determine a system’s behavior under both normal and at peak conditions. It helps to identify the maximum operating capacity of an application as well as any bottlenecks and determine which element is causing degradation. Activities of Load Testing might include: Validate that the system performs as expected when concurrent users access the application and gets the expected response time. This test is repeated with multiple users to get the response time and throughput. At the time of testing, the data base should be realistic. The test should be conducted on a dedicated server which stimulates the actual environment.
  • 16. Types of Non-Functional Testing Performance testing: Performance testing is performed to determine how fast some aspect of a system performs under a particular workload. It can serve different purposes like it can demonstrate that the system meets performance criteria. It can compare two systems to find which performs better. Or it can measure what part of the system or workload causes the system to perform badly. Activities in Performance Testing might include: Validate that the system meets the expected response time. Evaluate that the significant elements of the application meets the desired response time. It can also be conducted as a pert of integration testing. It can also be conducted as a part of systems testing.
  • 17. Types of Non-Functional Testing Compatibility testing: Compatibility testing is basically the testing of the application or the product built with the computing environment. It tests whether the application or the software product built is compatible with the hardware, operating system, database or other system software or not. Activities in Compatibility Testing might include: Test with each hardware with minimum and maximum configuration. Test with different browsers. Test cases are the same which were executed during functional testing. In case the number of hardware and software are too many, we can use OATS techniques to arrive at the test cases to have maximum coverage.
  • 18. Types of Non-Functional Testing Security testing: Security testing is basically to check that whether the application or the product is secured or not. Can anyone came tomorrow and hack the system or login the application without any authorization? It is a process to determine that an information system protects data and maintains functionality as intended. Scalability testing: It is the testing of a software application for measuring its capability to scale up in terms of any of its non-functional capability like load supported, the number of transactions, the data volume etc.
  • 19. Types of Non-Functional Testing Volume testing: Volume testing refers to testing a software application or the product with a certain amount of data. E.g., if we want to volume test our application with a specific database size, we need to expand our database to that size and then test the application’s performance on it. Some key points of Volume Testing: Subject the software to large amounts of data and check the limit where the software fails. Maximum database size is created and multiple client query the database or create larger report. If the application is processing database to create a report, a volume test would be to use a large result set and check that the report is printed correctly.
  • 20. Types of Non-Functional Testing Stress Testing: It is a form of testing that is used to determine the stability of a given system. It puts greater emphasis on robustness, availability, and error handling under a heavy load, rather than on what would be considered correct behavior under normal circumstances. The goals of such tests may be to ensure the software does not crash in conditions of insufficient computational resources (such as memory or disk space). Activities of Stress Testing might include: Test on low memory or low disc space on clients or servers that reveals the defects which cannot be found under normal conditions. Multiple user performance the same transactions on the same data. Multiple clients connected to servers with different workloads. Reduce the Think Time to “Zero” to stress the servers to their maximum stress.
  • 21. Types of Non-Functional Testing Recovery testing: Recovery testing is done in order to check how fast and better the application can recover after it has gone through any type of crash or hardware failure (etc). Recovery testing is the forced failure of the software in a variety of ways to verify that recovery is properly performed. Recovery Testing activities might include: Power interruption, to the client while doing CURD activities. Invalid database pointers and keys Database process are aborted or prematurely terminated. Database pointers, fields and keys are corrupted manually and directly within database. Physically disconnect the communication wires, power turn off, turn down the routers and network servers.
  • 22. Types of Non-Functional Testing Internationalization testing: Internationalization is a process of designing a software application so that it can be adapted to various languages and regions without any changes. Localization Testing: Localization is a process of adapting internationalized software for a specific region or language by adding local specific components and translating text.
  • 23. Estimating Non-Functional Requirements Once the team accepts a non-functional requirement into the project they need to remain in compliance with that non-functional requirement for the remainder of the project. Doing performance testing (or staying in compliance with any non-functional requirement) creates some amount of overhead on the team. This overhead, or tax, must be paid regularly. In some cases, the team and product owner will decide the tax must be paid every sprint. In other cases the team and product owner may agree to pay the tax every few sprints. Estimate the cost of initial compliance just like any other user story or product backlog item. The team and product owner will need to incorporate an estimate of when they'll do this. Adding performance testing after five sprints is different than adding it after 20 sprints. [5]
  • 24. Distinction between Functional and Non Functional Testing Functional testing ensures that your product meets customer and business requirements, and doesn’t have any major bugs. Functional testing is designed to determine that the application’s features and operations perform the way they should. Non-functional testing wants to see if the product stands up to customer expectations. Non-functional testing wants to know that the product “behaves” correctly. Failure to release a working product that meets the needs of consumer demands can damage your company’s reputation and reduce overall product sales. [6]
  • 25. Functional Testing The following list has been extracted from here [7]. Tests how well the system performs. Based on client requirements. Means testing the application against business requirements. It is a part of System Testing. Validates the behavior of the application.
  • 26. Non-Functional Testing The following list has been extracted from here [7]. Tests how well the system responds. Based on client expectations. Means testing the application against the client and performance requirements. Also a part of System Testing. Validates the performance of the application.
  • 27. References [1] https://en.wikipedia.org/wiki/Non-functional_testing [2] http://reqtest.com/testing-blog/functional-vs-non-functional-testing/ [3] http://www.softwaretestinghelp.com/what-is-non-functional-testing/ [4] https://www.mountaingoatsoftware.com/blog/non-functional-requirements- as-user-stories [5] https://www.mountaingoatsoftware.com/blog/estimating-non-functional- requirements [6] http://www.ibeta.com/functional-vs-non-functional-testing-whats-difference/ [7] http://testingbasicinterviewquestions.blogspot.com.br/2015/02/functional- testing-vs-non-functional-testing.html
  • 28. Next Steps Getting Started Guide GETTING STARTED RESOURCES FREE TRIAL Documentation Support Tutorials What we're reading this week Blog Start you 30-day free trial now