Project Management Methodology and Testing
The BITTER FRUIT of Testing
Testing can be a bitter fruit to swallow as it is repetitive and takes a long time. However it is critical for success.
Read the guidelines below then click these links below to learn more.
Builds/Packaging
Installs/Deployment
Test Planning and Scripting
Training Materials, Start-Up
Educate Test teams
Re-planning
Functional Testing/System Testing
Regression Testing
User test/ Acceptance test
Integration Testing/Systems/Performance testing/Capacity Testing/Concurrency testing
Training/ Educate End Users
Testing Guidelines
It is never too late to start writing test scripts.There are usually many pressures to move from development to testing, these are not usually cost effective, unless your development team is telling you that more than 100% of development has been completed. If you are worried what the development team might deliver then the specifications probably were not good enough, and anyway you can choose to review and have demonstrations from the team of development so far. Having a pre-testing delivery "sense check" element in your testing plans allows you not to move into a full and expensive system testing round prematurely.
Maybe before this sense checking also start with the developers demonstrating their application is ready for test.
Across this part of project delivery a good team ethic has to be present.
A set of testing plans should start with a full list of tests, some of which should be marked as regressive. These regressive tests include all the tests you feel should be tested if another release of the software is made. These items can be slowly removed when your are confident, that an area of the system will not be changed (Where specific tests can then be introduced.
Many tools exists to store and manage testing scripts, use one if your testing plans call for a testing period over four weeks. They can be expensive but will save you costs further down the line.
Iterative testing cycles can be frustrating so ensure the causes of this are tackled ahead of testing.
The quality of a release you will receive into system testing includes;
- Dependencies on quality of specifications
- Dependencies on on connection speeds to the development environment and application
- Dependencies on speed and quality of developers
- Dependencies on experience of testing team
- Dependencies on complexity of delivery
Ensure each is checked going into and during development.
Bear in mind a great deal of time will be allocated to testing so during any phase ensure you make completion targets in terms of due dates so that there are opportunities to increase resource if its taking longer than first planned.
Where possible each different type of test should be performed by different resources as different types of people will have different views on testing so your testing coverage will be greater.
Below are examples of the different types of tests involved in different phases of testing to demonstrate the objective and the different types of individual tests involved in each stage. This is of course an example only and real scripts will include many more tests (as many as you can possibly think of).
The example uses a development where an application is being made to allow input of a Name and Address into a database with Outbound and Inbound update to an external emails contacts system (e.g. interfacing with Gmail).
Note: All examples below are for demonstrating the differences between types of test only and are not necessarily the tests which would actually be present for the example given above.
Unit Test Scripts
Objective: To ensure the application being delivered from development into testing is of a high quality and is efficient and maintainable.
Note: Unit testing is generally on a component by component, method by method basis and much more detailed than described here.
Note: A mixture of internal checks to the program, and more demonstrable test.
Ref | Test | Expected Result | Pass/Fail | Comments |
UT1 | Ensure validation routines returns a 'Pass' or 'Fail' in any circumstance and 'Fail' contains a message. | Routine only returns Pass or Fail. All Fails have a message. |
|
|
UT2 | Ensure validation routine returns 'P'ass and no message is returned if blank is passed in house number. | Assert validation flag is set to "P" and no message is passed. |
|
|
UT3 | Input blank to house number via screen | Screen correctly calls validation routines. Validation routines do not fail house number validation or pass error back to screen. |
|
|
UT4 | Input 100 to house number via screen | Screen correctly calls validation routines. Validation routines do not fail house number validation or pass error back to screen. |
|
|
UT5 | Input 'abc' to house number via screen | Is not allowed to be entered in field |
|
|
UT6 | Input 'abc' to house number via database call | Screen correctly calls validation routines. Validation routines fail house number validation and pass error back to screen. |
|
|
Installation Test Scripts
Objective: Ensure the application as delivered can be installed
Ref | Test | Expected Result | Pass/Fail | Comments |
IN1 | Run installation according to installation instructions | Program is installed |
|
|
IN2 | Run up the program | Program runs |
|
|
Operations Testing/Smoke Testing
Objective: Ensure the program is able to be accessed for Systems/Functional Testing (Nothing much worse than handing something over for testing to find the application does not run).
Ref | Test | Expected Result | Pass/Fail | Comments |
OP1 | Run up program | Splash screen is displayed then initial form. |
|
|
OP2 | Input a name via the application, exit come back is the name accessible from the same application | Name is accessible after exiting then re-entering program. |
|
|
OP3 | Input a name via the application, exit come back is the name accessible from the email client | Name is accessible after exiting then re-entering email client. |
|
|
Functional/System Testing
Demonstrable elements of unit testing.
These tests should be documented before the unit testing scripts, as a domain expert writes the specification and these can then also be used as a basis for unit testing). They should then be again reviewed near the end of development to include any changes to specification which may have been made over the duration of the development.
Good system testers will also be able to read code/unit tests and add smart tests post the unit testing phase. It is worth spending money on getting a good system tester who will cover additionally tests from other areas as part of his/her own tests for functional test).
Back up the functional test scripts with screen prints/report layouts/wireframes to avoid any ambiguity
Functional test scripts can be split into Sense Checks (which can be repeated on future rounds of testing but where more detailed tests are required for a change but need to be supplemented with some more general testing, similar but more detailed than Operational Tests) and Full Checks which test every individual point against a specification.
Testing referenced should be cross referenced with specification points, in the specification.
Ref | Test | Expected Result | Pass/Fail | Comments |
FT1 | Input a name via the application, exit come back is the name accessible from the same application | Name is still assessible after exiting and re-entering program. |
|
|
FT2 | Input a name via the application, exit come back is the name accessible from email client | Name is still accessible after existing and re-entering email client |
|
|
FT3 | Input blank to house number via screen | Entry is allowed and no error messages are generated |
|
|
FT4 | Input 100 to house number via screen | Entry is accepted and no error message is given |
|
|
FT5 | Input abc to house number via screen | Error is issued and contact is not allowed to be saved. |
|
|
FT6 | Ensure the correct people or roles can access the application and that other peoples contacts are view only and cannot be edited. Login as A.N.Invalid pw 123 | User is not allowed to access application |
|
|
End to End /Integration Test Scripts
Objective: Check that existing, already developed, systems allow data to flow seamlessly back and forth with the new application and that security is reflected correctly in the target environment. Can be started in parallel with functional system testing, and must be allowed to complete on final version.
Ref | Test | Expected Result | Pass/Fail | Comments |
EE1 | Change a name in Outlook, then check the name in the database application has the change been reflected. | Name change is reflected |
|
|
EE2 | Ensure the correct people or roles can access the application and that other people's contacts are view only and cannot be edited. |
|
|
|
User Acceptance Test Scripts (UAT)
Objective: Allow the end users to confirm they are happy the new system is easy to use and meets the objectives and requirements.
Note: Scripting can start immediately in-line with the functional testing scripts, but need to be revisited 80% though system testing to pick up any changes which may have been made.
Note: You should include scripted tests and then also add an additional session where the user invents some of their own tests in the same style as they have had provided, to allow them to report any issues in areas you may not have considered .
See our User Acceptance Test guidelines checklistfor more information.
Ref | Test | Expected Result | Pass/Fail | Comments |
UAT1 | Input a name and address to database, save then revisit is all information retrieved. | All information is retrieved and process was straight forward |
|
|
|
|
|
|
|
User Defined Tests |
|
| ||
|
|
|
|
|
UAT |
|
|
|
|
UAT |
|
|
|
|
Performance and Capacity Testing
Objective: Ensure the target system can handle load.
Note: Functional and end to end testing has to complete before these tests are ran, a run can be made near end of these tests to prove the scripts.
Ref | Test | Expected Result | Pass/Fail | Comments |
PT1 | Input 100 Names with their addresses as quickly as possibly | Names Appear in Outlook within 1 minute |
|
|
FT2 | Change 100 Addresses in Outlook | Addressed synchronised back to outlook within 1 minute |
|
|
Testing Estimation
Below is an example of the type of estimates which might be made for the various phases of testing for a development.The estimates assume a 10 day development with 2 days of additional unit testing. The percentages shown show give an idea of what percentage of the development estimate should be used to allocate to testing time for a development of 12 days.
Again this is for example only, estimates for the testing of piece of development have many dependencies (see above) and many variables. It is not really possible to tell how many issues will be raised unless you do similar pieces of development with similar people all the time. Estimating on the high side will give you contingency for things going wrong, or enable you to add additional tests, so is advised. As a rule of thumb the larger the development the more issues you will have, the larger the development the lower the percentage of the total development time you will need to spend on system testing.
The estimates below exclude an element of time for Project Management, Planning, Status Reporting, Meetings, Training. If you are making your own estimates remember to add these and remember to consider the amount of people you will be using for the activities. (using two people obviously doubles the estimate etc).
Preparation | Test Running | Contingency/ | Contingency/ | Contingency/ | % of Development | Notes: | |
Installation Testing | 0.10 | 0.06 | 2 | ||||
Operations Testing/Smoke Testing | 0.13 | 0.13 | 0.13 | 0.13 | 0.13 | 6 | |
System Testing | 2.50 | 5.00 | 2.50 | 1.25 | 0.63 | 119 | The smaller the development the larger the percentage |
Integration/End To End Testing | 0.75 | 0.50 | 0.25 | 0.13 | 0.06 | 17 | Where applicable. |
User Acceptance Testing | 1.00 | 0.50 | 15 | ||||
Performance Testing | 0.50 | 0.25 | 8 | The smaller the development the smaller the percentage. | |||
| |||||||
Total Days | 4.88 | 11.35 | 2.88 | 1.50 | 0.81 | ||
% of Development | 49 | 114 | 29 | 15 | 8 | ||
% of Previous Test Cycle | 25 | 52 | 54 |
Be a pioneer and write the first comment.
Note: Current average rating of based on reviews and ratings. (1-Low, 5-High)
Note: Comments and ratings help this site get better; if you see something missing, see something wrong, have a question, or want to suggest something to improve then comment below and join the dialogue;