Performance Testing - Stress testing 2
Stress testing :is subjecting a system to an unreasonable load while denying it the resources (e.g., RAM, disc, mips, interrupts, etc.) needed to process that load. The idea is to stress a system to the breaking point in order to find bugs that will make that break potentially harmful. The system is not expected to process the overload without adequate resources, but to behave (e.g., fail) in a decent manner (e.g., not corrupting or losing data). Bugs and failure modes discovered under stress testing may or may not be repaired depending on the application, the failure mode, consequences, etc. The load (incoming transaction stream) in stress testing is often deliberately distorted so as to force the system into resource depletion.
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about software testing.
http://www.crestech.in
http://www.crestechsoftware.com.au
Monday, June 29, 2009
Why White box testing when black box testing is there to test conformance to requirements
Why White box testing when black box testing is there to test conformance to requirements
Logic errors and incorrect assumptions most likely to be made when coding for "special cases". Need to ensure these execution paths are tested.
May find assumptions about execution paths incorrect, and so make design errors. White box testing can find these errors.
Typographical errors are random. Just as likely to be on an obscure logical path as on a mainstream path.
Software Testing Training
Software testing institute
corporate training software testing
Logic errors and incorrect assumptions most likely to be made when coding for "special cases". Need to ensure these execution paths are tested.
May find assumptions about execution paths incorrect, and so make design errors. White box testing can find these errors.
Typographical errors are random. Just as likely to be on an obscure logical path as on a mainstream path.
Software Testing Training
Software testing institute
corporate training software testing
Regression Testing
Regression Testing
Testing conducted for the purpose of evaluating whether or not a change to the system (all CM items) has introduced a new failure. Regression testing is often accomplished through the construction, execution and analysis of product and system tests.
Regression testing is an expensive but necessary activity performed on modified software to provide confidence that changes are correct and do not adversely affect other system components. Four things can happen when a developer attempts to fix a bug. Three of these things are bad, and one is good:
Because of the high probability that one of the bad outcomes will result from a change to the system, it is necessary to do regression testing.
It can be difficult to determine how much re-testing is needed, especially near the end of the development cycle. Most industrial testing is done via test suites; automated sets of procedures designed to exercise all parts of a program and to show defects. While the original suite could be used to test the modified software, this might be very time-consuming. A regression test selection technique chooses, from an existing test set, the tests that are deemed necessary to validate modified software.
There are three main groups of test selection approaches in use:
Minimization approaches seek to satisfy structural coverage criteria by identifying a minimal set of tests that must be rerun.
Coverage approaches are also based on coverage criteria, but do not require minimization of the test set. Instead, they seek to select all tests that exercise changed or affected program components.
Safe attempt instead to select every test that will cause the modified program to produce different output than original program.
An interesting approach to limiting test cases is based on whether we can confine testing to the "vicinity" of the change. (Ex. If I put a new radio in my car, do I have to do a complete road test to make sure the change was successful?) A new breed of regression test theory tries to identify, through program flows or reverse engineering, where boundaries can be placed around modules and subsystems. These graphs can determine which tests from the existing suite may exhibit changed behavior on the new version.
Regression testing has been receiving more attention as corporations focus on fixing the 'Year 2000 Bug'. The goal of most Y2K is to correct the date handling portions of their system without changing any other behavior. A new 'Y2K' version of the system is compared against a baseline original system. With the obvious exception of date formats, the performance of the two versions should be identical. This means not only do they do the same things correctly, they also do the same things incorrectly. A non-Y2K bug in the original software should not have been fixed by the Y2K work.
A frequently asked question about regression testing is 'The developer says this problem is fixed. Why do I need to re-test?’ to which the answer is 'The same person probably told you it worked in the first place'
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about software testing.
http://www.crestech.in
http://www.crestechsoftware.com.au
Testing conducted for the purpose of evaluating whether or not a change to the system (all CM items) has introduced a new failure. Regression testing is often accomplished through the construction, execution and analysis of product and system tests.
Regression testing is an expensive but necessary activity performed on modified software to provide confidence that changes are correct and do not adversely affect other system components. Four things can happen when a developer attempts to fix a bug. Three of these things are bad, and one is good:
Because of the high probability that one of the bad outcomes will result from a change to the system, it is necessary to do regression testing.
It can be difficult to determine how much re-testing is needed, especially near the end of the development cycle. Most industrial testing is done via test suites; automated sets of procedures designed to exercise all parts of a program and to show defects. While the original suite could be used to test the modified software, this might be very time-consuming. A regression test selection technique chooses, from an existing test set, the tests that are deemed necessary to validate modified software.
There are three main groups of test selection approaches in use:
Minimization approaches seek to satisfy structural coverage criteria by identifying a minimal set of tests that must be rerun.
Coverage approaches are also based on coverage criteria, but do not require minimization of the test set. Instead, they seek to select all tests that exercise changed or affected program components.
Safe attempt instead to select every test that will cause the modified program to produce different output than original program.
An interesting approach to limiting test cases is based on whether we can confine testing to the "vicinity" of the change. (Ex. If I put a new radio in my car, do I have to do a complete road test to make sure the change was successful?) A new breed of regression test theory tries to identify, through program flows or reverse engineering, where boundaries can be placed around modules and subsystems. These graphs can determine which tests from the existing suite may exhibit changed behavior on the new version.
Regression testing has been receiving more attention as corporations focus on fixing the 'Year 2000 Bug'. The goal of most Y2K is to correct the date handling portions of their system without changing any other behavior. A new 'Y2K' version of the system is compared against a baseline original system. With the obvious exception of date formats, the performance of the two versions should be identical. This means not only do they do the same things correctly, they also do the same things incorrectly. A non-Y2K bug in the original software should not have been fixed by the Y2K work.
A frequently asked question about regression testing is 'The developer says this problem is fixed. Why do I need to re-test?’ to which the answer is 'The same person probably told you it worked in the first place'
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about software testing.
http://www.crestech.in
http://www.crestechsoftware.com.au
Performance Testing - Volume Testing
Performance Testing - Volume Testing
Volume Testing: Testing where the system is subjected to large volumes of data. As regards faults that should be found through volume testing are those, where the behaviour of the software deviates from that expected for a specified volume of data. Thus a bank system will be tested for faults at much larger volumes of data, than that of small retailer software. A fault which is only manifested on a table with a million records will be of no concern to the retail software, but will be picked up by the bank testers.
For example, a window often used window object is populated with data, by calling a database object which runs a complex SQL query. Supposing the component is tested against tables with only 4-5 records. Of course it will return within seconds. Everything seems fine. It is then integrated with the window, and system tested. Again everything seems ok. It is only when the application is in User Acceptance (or even gone live) and it is tested against 100,000 records, is it discovered that, the SQL was not properly optimized and the tables not indexed. Thus it should have been tested at the component level.
Volume testing needs two things. Firstly clear expected outcomes of how the software is to behave for a given level of data. Secondly, data, and lots of it.
The expected behaviour at various levels, should be in the specification documentation. Ideally this will say something like "the customers details will be returned returned on the screen within 3 seconds, from a database with 1 million customer records." This gives the tester a benchmark to base a test case on.
The second requirement for data, needs either real life data, or simulated data. Usually, real life data will come in the form of a customer database, that has had private information, such as names and account numbers scrambled. Alternatively records can be created from scratch using automated tools or by adding rules directly on to the database, with SQL.
As with all testing, proper records must be kept showing the inputs, outputs other information, to aid potential debugging and audit purposes
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about software testing.
http://www.crestech.in
http://www.crestechsoftware.com.au
Volume Testing: Testing where the system is subjected to large volumes of data. As regards faults that should be found through volume testing are those, where the behaviour of the software deviates from that expected for a specified volume of data. Thus a bank system will be tested for faults at much larger volumes of data, than that of small retailer software. A fault which is only manifested on a table with a million records will be of no concern to the retail software, but will be picked up by the bank testers.
For example, a window often used window object is populated with data, by calling a database object which runs a complex SQL query. Supposing the component is tested against tables with only 4-5 records. Of course it will return within seconds. Everything seems fine. It is then integrated with the window, and system tested. Again everything seems ok. It is only when the application is in User Acceptance (or even gone live) and it is tested against 100,000 records, is it discovered that, the SQL was not properly optimized and the tables not indexed. Thus it should have been tested at the component level.
Volume testing needs two things. Firstly clear expected outcomes of how the software is to behave for a given level of data. Secondly, data, and lots of it.
The expected behaviour at various levels, should be in the specification documentation. Ideally this will say something like "the customers details will be returned returned on the screen within 3 seconds, from a database with 1 million customer records." This gives the tester a benchmark to base a test case on.
The second requirement for data, needs either real life data, or simulated data. Usually, real life data will come in the form of a customer database, that has had private information, such as names and account numbers scrambled. Alternatively records can be created from scratch using automated tools or by adding rules directly on to the database, with SQL.
As with all testing, proper records must be kept showing the inputs, outputs other information, to aid potential debugging and audit purposes
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about software testing.
http://www.crestech.in
http://www.crestechsoftware.com.au
Performance Testing - Load testing
Performance Testing - Load testing
Load testing: is subjecting a system to a statistically representative (usually) load. The two main reasons for using such loads is in support of software reliability testing and in performance testing. The term "load testing" by itself is too vague and imprecise to warrant use. For example, do you mean representative load," "overload," "high load," etc. In performance testing, load is varied from a minimum (zero) to the maximum level the systemcan sustain without running out of resources or having, transactions suffer (application-specific) excessive delay.
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about software testing.
http://www.crestech.in
http://www.crestechsoftware.com.au
Load testing: is subjecting a system to a statistically representative (usually) load. The two main reasons for using such loads is in support of software reliability testing and in performance testing. The term "load testing" by itself is too vague and imprecise to warrant use. For example, do you mean representative load," "overload," "high load," etc. In performance testing, load is varied from a minimum (zero) to the maximum level the systemcan sustain without running out of resources or having, transactions suffer (application-specific) excessive delay.
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about software testing.
http://www.crestech.in
http://www.crestechsoftware.com.au
Performance Testing - Load testing
Performance Testing - Load testing
Load testing: is subjecting a system to a statistically representative (usually) load. The two main reasons for using such loads is in support of software reliability testing and in performance testing. The term "load testing" by itself is too vague and imprecise to warrant use. For example, do you mean representative load," "overload," "high load," etc. In performance testing, load is varied from a minimum (zero) to the maximum level the systemcan sustain without running out of resources or having, transactions suffer (application-specific) excessive delay.
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about software testing.
http://www.crestech.in
http://www.crestechsoftware.com.au
Load testing: is subjecting a system to a statistically representative (usually) load. The two main reasons for using such loads is in support of software reliability testing and in performance testing. The term "load testing" by itself is too vague and imprecise to warrant use. For example, do you mean representative load," "overload," "high load," etc. In performance testing, load is varied from a minimum (zero) to the maximum level the systemcan sustain without running out of resources or having, transactions suffer (application-specific) excessive delay.
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about software testing.
http://www.crestech.in
http://www.crestechsoftware.com.au
Integration Testing
Integration Testing
One of the most difficult aspects of software development is the integration and testing of large, untested sub-systems. The integrated system frequently fails in significant and mysterious ways, and it is difficult to fix it
Integration testing exercises several units that have been combined to form a module, subsystem, or system. Integration testing focuses on the interfaces between units, to make sure the units work together. The nature of this phase is certainly 'white box', as we must have a certain knowledge of the units to recognize if we have been successful in fusing them together in the module.
There are three main approaches to integration testing: top-down, bottom-up and 'big bang'. Top-down combines, tests, and debugs top-level routines that become the test 'harness' or 'scaffolding' for lower-level units. Bottom-up combines and tests low-level units into progressively larger modules and subsystems. 'Big bang' testing is, unfortunately, the prevalent integration test 'method'. This is waiting for all the module units to be complete before trying them out together.
Integration tests can rely heavily on stubs or drivers. Stubs stand-in for finished subroutines or sub-systems. A stub might consist of a function header with no body, or it may read and return test data from a file, return hard-coded values, or obtain data from the tester. Stub creation can be a time consuming piece of testing.
The cost of drivers and stubs in the top-down and bottom-up testing methods is what drives the use of 'big bang' testing. This approach waits for all the modules to be constructed and tested independently, and when they are finished, they are integrated all at once. While this approach is very quick, it frequently reveals more defects than the other methods. These errors have to be fixed and as we have seen, errors that are found 'later' take longer to fix. In addition, like bottom up, there is really nothing that can be demonstrated until later in the process.
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about software testing.
http://www.crestech.in
http://www.crestechsoftware.com.au
One of the most difficult aspects of software development is the integration and testing of large, untested sub-systems. The integrated system frequently fails in significant and mysterious ways, and it is difficult to fix it
Integration testing exercises several units that have been combined to form a module, subsystem, or system. Integration testing focuses on the interfaces between units, to make sure the units work together. The nature of this phase is certainly 'white box', as we must have a certain knowledge of the units to recognize if we have been successful in fusing them together in the module.
There are three main approaches to integration testing: top-down, bottom-up and 'big bang'. Top-down combines, tests, and debugs top-level routines that become the test 'harness' or 'scaffolding' for lower-level units. Bottom-up combines and tests low-level units into progressively larger modules and subsystems. 'Big bang' testing is, unfortunately, the prevalent integration test 'method'. This is waiting for all the module units to be complete before trying them out together.
Integration tests can rely heavily on stubs or drivers. Stubs stand-in for finished subroutines or sub-systems. A stub might consist of a function header with no body, or it may read and return test data from a file, return hard-coded values, or obtain data from the tester. Stub creation can be a time consuming piece of testing.
The cost of drivers and stubs in the top-down and bottom-up testing methods is what drives the use of 'big bang' testing. This approach waits for all the modules to be constructed and tested independently, and when they are finished, they are integrated all at once. While this approach is very quick, it frequently reveals more defects than the other methods. These errors have to be fixed and as we have seen, errors that are found 'later' take longer to fix. In addition, like bottom up, there is really nothing that can be demonstrated until later in the process.
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about software testing.
http://www.crestech.in
http://www.crestechsoftware.com.au
Functional testing
Functional testing
Functional testing: Application of test data derived from the specified functional requirements without regard to the final program structure. Also known as black-box testing.
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about software testing.
http://www.crestech.in
http://www.crestechsoftware.com.au
Functional testing: Application of test data derived from the specified functional requirements without regard to the final program structure. Also known as black-box testing.
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about software testing.
http://www.crestech.in
http://www.crestechsoftware.com.au
Exploratory Testing
Exploratory Testing
An interactive process of concurrent product exploration, test design, and test execution. The heart of exploratory testing can be stated simply: The outcome of this test influences the design of the next test.
The plainest definition of exploratory testing is test design and test execution at the same time. This is the opposite of scripted testing (predefined test procedures, whether manual or automated). Exploratory tests, unlike scripted tests, are not defined in advance and carried out precisely according to plan. This may sound like a straightforward distinction, but in practice it's murky. That's because "defined" is a spectrum. Even an otherwise elaborately defined test procedure will leave many interesting details (such as how quickly to type on the keyboard, or what kinds of behavior to recognize as a failure) to the discretion of the tester. Likewise, even a free-form exploratory test session will involve tacit constraints or mandates about what parts of the product to test, or what strategies to use. A good exploratory tester will write down test ideas and use them in later test cycles. Such notes sometimes look a lot like test scripts, even if they aren't. Exploratory testing is sometimes confused with "ad hoc" testing. Ad hoc testing normally refers to a process of improvised, impromptu bug searching. By definition, anyone can do ad hoc testing
What kinds of specifics affect ET? Here are some of them:
1. the mission of the test project
2. the mission of this particular test session
3. the role of the tester
4. the tester (skills, talents, and preferences)
5. available tools and facilities
6. available time
7. available test data and materials
8. available help from other people
9. accountability requirements
10. what the tester‘s clients care about
11. the current testing strategy
12. the status of other testing efforts on the same product
13. the product, itself- its user interface - its behavior - its present state of execution - its defects- its testability- its purpose
14. what the tester knows about the product- what just happened in the previous test - known problems with it- past problems with it - strengths and weaknesses - risk areas and magnitude of perceived risk - recent changes to it - direct observations of it- rumors about it - the nature of its users and user behavior - how it‘s supposed to work - how it‘s put together - how it‘s similar to or different from other products
15. what the tester would like to know about the product
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about software testing.
http://www.crestech.in
http://www.crestechsoftware.com.au
An interactive process of concurrent product exploration, test design, and test execution. The heart of exploratory testing can be stated simply: The outcome of this test influences the design of the next test.
The plainest definition of exploratory testing is test design and test execution at the same time. This is the opposite of scripted testing (predefined test procedures, whether manual or automated). Exploratory tests, unlike scripted tests, are not defined in advance and carried out precisely according to plan. This may sound like a straightforward distinction, but in practice it's murky. That's because "defined" is a spectrum. Even an otherwise elaborately defined test procedure will leave many interesting details (such as how quickly to type on the keyboard, or what kinds of behavior to recognize as a failure) to the discretion of the tester. Likewise, even a free-form exploratory test session will involve tacit constraints or mandates about what parts of the product to test, or what strategies to use. A good exploratory tester will write down test ideas and use them in later test cycles. Such notes sometimes look a lot like test scripts, even if they aren't. Exploratory testing is sometimes confused with "ad hoc" testing. Ad hoc testing normally refers to a process of improvised, impromptu bug searching. By definition, anyone can do ad hoc testing
What kinds of specifics affect ET? Here are some of them:
1. the mission of the test project
2. the mission of this particular test session
3. the role of the tester
4. the tester (skills, talents, and preferences)
5. available tools and facilities
6. available time
7. available test data and materials
8. available help from other people
9. accountability requirements
10. what the tester‘s clients care about
11. the current testing strategy
12. the status of other testing efforts on the same product
13. the product, itself- its user interface - its behavior - its present state of execution - its defects- its testability- its purpose
14. what the tester knows about the product- what just happened in the previous test - known problems with it- past problems with it - strengths and weaknesses - risk areas and magnitude of perceived risk - recent changes to it - direct observations of it- rumors about it - the nature of its users and user behavior - how it‘s supposed to work - how it‘s put together - how it‘s similar to or different from other products
15. what the tester would like to know about the product
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about software testing.
http://www.crestech.in
http://www.crestechsoftware.com.au
Boundary Value Analysis
Boundary Value Analysis
Large number of errors tend to occur at boundaries of the input domain.
BVA leads to selection of test cases that exercise boundary values.
BVA complements equivalence partitioning. Rather than select any element in an equivalence class, select those at the ''edge' of the class.
Examples:
For a range of values bounded by a and b, test (a-1), a, (a+1), (b-1), b, (b+1).
If input conditions specify a number of values n, test with (n-1), n and (n+1) input values.
Apply 1 and 2 to output conditions (e.g., generate table of minimum and maximum size).
If internal program data structures have boundaries (e.g., buffer size, table limits), use input data to exercise structures on boundaries.
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about software testing.
http://www.crestech.in
http://www.crestechsoftware.com.au
Large number of errors tend to occur at boundaries of the input domain.
BVA leads to selection of test cases that exercise boundary values.
BVA complements equivalence partitioning. Rather than select any element in an equivalence class, select those at the ''edge' of the class.
Examples:
For a range of values bounded by a and b, test (a-1), a, (a+1), (b-1), b, (b+1).
If input conditions specify a number of values n, test with (n-1), n and (n+1) input values.
Apply 1 and 2 to output conditions (e.g., generate table of minimum and maximum size).
If internal program data structures have boundaries (e.g., buffer size, table limits), use input data to exercise structures on boundaries.
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about software testing.
http://www.crestech.in
http://www.crestechsoftware.com.au
About Software Testing
About Software Testing
Process of evaluation of Software application: To verify it satisfy the specified requirements With intent to find errors To Check if the system ?Fit for Purpose? Simulation of real time scenarios.
Testing is not a one-time activity—applications need to be tested throughout their lifecycle. Every version upgrade, module addition, or enhancement, as well as every implementation at a new site or increase in user load, needs to be put through comprehensive testing. The Program Doesn‘t Work Nobody would pay you to test if their program didn‘t have bugs. All programs have bugs. Any change to a program can cause new bugs, and any aspect of the program can be broken. You DON‘T ?verify that the program is working.? You FIND bugs. “If you set your mind to show that a program works correctly, you’ll be more likely to miss problems than if you want and expect the program to fail.” Complete Testing is Impossible:
1. There are a nearly infinite number of paths through any non-trivial program.
2. There are a virtually infinite set of combinations of data that you can feed the program.
You can’t test them all.
1. Therefore, your task is to find bugs --not to find all the bugs.
2. You want to find as many bugs as possible
3. Find the most serious bugs
4. Find bugs as early as possible
5. Your challenges will require judgment, trade-offs, and efficiency.
“Q: How many QA testers does it take to change a lightbulb?
A: QA testers don't change anything. They just report that it's dark.” Kerry Zallar?
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about software testing.
http://www.crestech.in
http://www.crestechsoftware.com.au
Process of evaluation of Software application: To verify it satisfy the specified requirements With intent to find errors To Check if the system ?Fit for Purpose? Simulation of real time scenarios.
Testing is not a one-time activity—applications need to be tested throughout their lifecycle. Every version upgrade, module addition, or enhancement, as well as every implementation at a new site or increase in user load, needs to be put through comprehensive testing. The Program Doesn‘t Work Nobody would pay you to test if their program didn‘t have bugs. All programs have bugs. Any change to a program can cause new bugs, and any aspect of the program can be broken. You DON‘T ?verify that the program is working.? You FIND bugs. “If you set your mind to show that a program works correctly, you’ll be more likely to miss problems than if you want and expect the program to fail.” Complete Testing is Impossible:
1. There are a nearly infinite number of paths through any non-trivial program.
2. There are a virtually infinite set of combinations of data that you can feed the program.
You can’t test them all.
1. Therefore, your task is to find bugs --not to find all the bugs.
2. You want to find as many bugs as possible
3. Find the most serious bugs
4. Find bugs as early as possible
5. Your challenges will require judgment, trade-offs, and efficiency.
“Q: How many QA testers does it take to change a lightbulb?
A: QA testers don't change anything. They just report that it's dark.” Kerry Zallar?
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about software testing.
http://www.crestech.in
http://www.crestechsoftware.com.au
Subscribe to:
Posts (Atom)