The following are the actives of the SDLC
Usability Engineering - Usability Testing
Usability Engineering, an empirical science has quite a simple definition. It studies the human interaction and cognitive behavior of an individual with respect to performing as task. It could be as simple as a driving a vehicle or using a product. Users interaction in performing a task should be in sync with the workflow of the product. Usability Engineering as a science helps in achieving this goal.
A Product should be usable. It means that people can use a product easily and efficiently to accomplish their own tasks. A product, which is usable, enables workers to concentrate on their tasks and to do real work, rather than on the tools they use to perform their tasks.
A usable product has the following characteristics:
• It’s easy to learn
• Efficient to use
• Provides quick recovery from errors
• Easy to remember
• Enjoyable to use
• Visually pleasing
Usability applies to every aspect of a product with which a person interacts (hardware, software, menus, icons, messages, documentation, training, and on-line help). Every design and development decision made throughout the product cycle has an impact on that product’s usability.
As customers depend more and more on software to get their jobs done and become more critical consumers, usability can be the critical factor that ensures that products will be used.
Usability engineering involves a variety of techniques that can provide important information about how customers work with your product. Different techniques are used at different stages of a product’s development.
For example, as processes are being engineered and requirements are being developed, observations and interviews may be the techniques of choice. Later in the development cycle, as the “look and feel” of a product is being designed, benchmarking, prototyping and participatory design may be useful techniques. Once a design has been determined, usability testing may be used more appropriately. Usability is an iterative process, just like software development. The usability process works best if it is done in partnership with product development.
1. User and task observations – observing users at their jobs, identifying their typical work tasks and procedures, analyzing their work processes, and understanding people in the context of their work.
2. Interviews, focus groups and questionnaires – meeting with users, finding out about their preferences, experiences and needs.
3. Benchmarking and competitive analysis – evaluating the usability of similar products in the marketplace.
4. Participatory design - participating in design and bringing the user’s perspective to the early stages of development.
5. Paper prototyping – including users early in the development process through prototypes prepared on paper before coding begins.
6. Creation of guidelines - helping to assure consistency in design through development of standards and guidelines.
7. Heuristic evaluations - evaluating software against accepted usability principles and making recommendations to enhance usability.
8. Usability testing - observing users performing real tasks with the application, recording what they do, analyzing the results and recommending appropriate changes.
Benefits of Usability: Usability engineering provides important benefits in terms of cost, product quality and customer satisfaction. It can improve development productivity through more efficient design and fewer code revisions. It can help to eliminate over-design by emphasizing the functionality required to meet the needs of real users. Design problems can be detected earlier in the development process, saving both time and money. It can provide further cost savings through reduced support costs, reduced training requirements and greater user productivity. A usable product means more satisfied customers and a better reputation for the product and for the organization that developed it.
Today many leading corporations are incorporating usability engineering into their product development cycles. For them, usability is becoming a competitive advantage.
A usability test is a formal evaluation process that has as its goal improvement of the usability of the product being tested. It differs from a quality assurance or quality test, which has as its goal assessing whether the product works according to specifications. It differs from a customer assurance test, a pilot test and a beta test because the usability test ensures the collection of systematic, recorded, quantifiable data and observation of behaviors.
A usability test has these five characteristics:
• Each test has specific goals and concerns that are tested.
• The participants represent real users (6 to 12 participants are typical).
• The participants do real tasks.
• The participants are observed and recorded.
• The data is analyzed, problems diagnosed and recommendations made.
• Planning the test, developing participant’s profiles, identifying participants from user pool, creating test materials, writing task scenarios and determining usability criteria and measures.
• Preparing the test location, pilot testing materials and procedures.
• Introducing the participant to the situation, the product and the procedure.
• Running of the task-based test, where participants are asked to complete a series of tasks that address the specific goals and concerns being tested.
• Participants are asked to “think aloud” (articulate their thoughts, feeling and actions). This data and the recorded video images, helps target areas that are confusing, unclear or misleading during the analysis stage.
• Debriefing the participant to get final thoughts, subjective feelings about the product and suggestions for improvement.
• Analyzing the data, making recommendations and documenting findings
Testing Your Project with end users that represent the end users of the final system. During the test, we can able to identify those areas of the system that will cause users to make errors, become less productive or be unable to complete a required activity.
The deliverable from a usability test is a report that details the problems encountered by the participants and recommendations for chance based on known human factors, cognitive and behavioral principles and recognized best practices.
Best Practices in Automated Testing
The Case for Automated Testing
Today, rigorous application testing is a critical part of virtually all software development projects. As more organizations develop mission – critical systems to support their business activities, the need is greatly increased for testing methods that support business objectives. It is necessary to ensure that these systems are reliable, built according to specification and have the ability to support business processes. Many internal and external factors are forcing organizations to ensure a high level of software quality and reliability.
In the past, most software tests were performed using manual methods. This required a large staff of test personnel to perform expensive and time-consuming manual test procedures. Owing to the size and complexity of today’s advanced software applications, manual testing is no longer a viable option for most testing situations.
The cost of performing manual testing is prohibitive when compared to automated methods. The reason is that computers can execute instructions many times faster and with fewer errors than individuals. Many automated testing tools can replicate the activity of a large number of users (and their associated transactions) using a single computer. Therefore, load/stress testing using automated methods requires only a fraction of the computer hardware that would be necessary to complete a manual test.
Automation allows the testing organization to perform consistent and repeatable test. When applications need to be deployed across different hardware or software platforms, standard or benchmark tests can be created and repeated on target platforms to ensure that new platforms operate consistently.
The productivity gains delivered by automated testing allow and encourage organization to test more often and more completely. Greater application test coverage also reduces the risk if exposing users to malfunctioning or non-compliant software.
Full-featured automated testing systems also produce convenient test reporting and analysis. These reports provide a standardized measure of test status and results, thus allowing more accurate interpretation of testing outcomes. Manual methods require the user to self-document test procedures and test results
Understanding the Testing Process
The introduction of automated testing into the business environment involves far more than buying and installing an automated testing tool.
Typical Testing Steps: Most software testing projects can be divided into general steps
Test Planning: This step determines like ‘which’ and ‘when’.
Test Design: This step determines how the tests should be built the level of quality.
Test Environment Preparation: Technical environment is established during this step.
Test Construction: At this step, test scripts are generated and test cases are developed.
Test Execution: This step is where the test scripts are executed according to the test plans.
Test evaluation: After the test is executed, the test results are compared to the expected results and evaluations can be made about the quality of an application.
Most, but not all, types of tests can be automated. Certain types of tests like user comprehension tests test that run only once and tests that require constant human intervention are usually not worth the investment incurred to automate. The following are examples of criteria that can be used to identify tests that are prime candidates for automation.
High path frequency – Automated testing can be used to verify the performance of application paths that are used with a high degree of frequency when the software is running in full production. Examples include: creating customer records.
Critical Business Processes – Mission-critical processes are prime candidates for automated testing. Examples include: financial month-end closings, production planning, sales order entry and other core activities. Any application with a high –degree of risk associated with a failure is a good candidate for test automation.
Repetitive Testing – If a testing procedure can be reused many times, it is also a prime candidate for automation
Applications with a Long Life Span – If an application is planned to be in production for a long period of time, the greater the benefits are from automation.
In performing software testing, there are many tasks that need to be performed before or after the actual test. For example, if a test needs to be executed to create sales orders against current inventory, goods need to be in inventory. The tasks associated with placing items in inventory can be automated so that the test can run repeatedly. Additionally, highly repetitive tasks not associated with testing can be automated utilizing the same approach.
There is no clear consensus in the testing community about which group within an organization should be responsible for performing the testing function. It depends on the situation prevailing in the organization.
Life Cycle of Testing Process
Once the project is confirmed to start, the phases of the development of project can be divided into the following phases:
· Software requirements phase.
· Software Design
· Implementation
· Testing
· Maintenance
In the whole development process, testing consumes highest amount of time. But most of the developers oversee that and testing phase is generally neglected. As a consequence, erroneous software is released. The testing team should be involved right from the requirements stage itself.
The various phases involved in testing, with regard to the software development life cycle are:
1. Requirements stage
2. Test Plan
3. Test Design.
4. Design Reviews
5. Code Reviews
6. Test Cases preparation.
7. Test Execution
8. Test Reports.
9. Bugs Reporting
10. Reworking on patches.
11. Release to production.
Normally in many companies, developers itself take part in the requirements stage. Especially for product-based companies, a tester should also be involved in this stage. Since a tester thinks from the user side whereas a developer can’t. A separate panel should be formed for each module comprising a developer, a tester and a user. Panel meetings should be scheduled in order to gather everyone’s view. All the requirements should be documented properly for further use and this document is called “Software Requirements Specifications”.
Without a good plan, no work is a success. A successful work always contains a good plan. The testing process of software should also require good plan. Test plan document is the most important document that brings in a process – oriented approach. A test plan document should be prepared after the requirements of the project are confirmed. The test plan document must consist of the following information:
• Total number of features to be tested.
• Testing approaches to be followed.
• The testing methodologies
• Number of man-hours required.
• Resources required for the whole testing process.
• The testing tools that are to be used.
• The test cases, etc
Test Design is done based on the requirements of the project. Test has to be designed based on whether manual or automated testing is done. For automation testing, the different paths for testing are to be identified first. An end to end checklist has to be prepared covering all the features of the project.
The test design is represented pictographically. The test design involves various stages. These stages can be summarized as follows:
• The different modules of the software are identified first.
• Next, the paths connecting all the modules are identified.
Then the design is drawn. The test design is the most critical one, which decides the test case preparation. So the test design assesses the quality of testing process.
Test cases should be prepared based on the following scenarios:
• Positive scenarios
• Negative scenarios
• Boundary conditions and
• Real World scenarios
Design Reviews
The software design is done in systematical manner or using the UML language. The tester can do the reviews over the design and can suggest the ideas and the modifications needed.
Code reviews are similar to unit testing. Once the code is ready for release, the tester should be ready to do unit testing for the code. He must be ready with his own unit test cases. Though a developer does the unit testing, a tester must also do it. The developers may oversee some of the minute mistakes in the code, which a tester may find out.
Once the unit testing is completed and the code is released to QA, the functional testing is done. A top-level testing is done at the beginning of the testing to find out the top-level failures. If any top-level failures occur, the bugs should be reported to the developer immediately to get the required workaround.
The test reports should be documented properly and the bugs have to be reported to the developer after the testing is completed.
Automated Testing Advantages, Disadvantages and Guidelines
Minimally, such a process includes:
Detailed test cases, including predictable "expected results", which have been developed from Business Functional Specifications and Design documentation
A standalone Test Environment, including a Test Database that is restorable to a known constant, such that the test cases are able to be repeated each time there are modifications made to the application.
Functional - testing that operations perform as expected.
Regression - testing that the behavior of the system has not changed.
Exception or Negative - forcing error conditions in the system.
Stress - determining the absolute capacities of the application and operational infrastructure.
Performance - providing assurance that the performance of the system will be adequate for both batch runs and online transactions in relation to business projections and requirements.
Load - determining the points at which the capacity and performance of the system become degraded to the situation that hardware or software upgrades would be required.
Reliable: Tests perform precisely the same operations each time they are run, thereby eliminating human error
Repeatable: You can test how the software reacts under repeated execution of the same operations.
Programmable: You can program sophisticated tests that bring out hidden information from the application.
Comprehensive: You can build a suite of tests that covers every feature in your application.
Reusable: You can reuse tests on different versions of an application, even if the user interface changes.
Better Quality Software: Because you can run more tests in less time with fewer resources
Fast: Automated Tools run tests significantly faster than human users.
Cost Reduction: As the number of resources for regression test are reduced.
Choosing the right tools for the job and targeting the right areas of the organization to deploy them can only realize these benefits. The right areas where the automation fit must be chosen.
1. Highly redundant tasks or scenarios
2. Repetitive tasks that are boring or tend to cause human error
3. Well-developed and well-understood use cases or scenarios first
4. Relatively stable areas of the application over volatile ones must be automated.
· • Concise: As simple as possible and no simpler.
· • Self-Checking: Test reports its own results; needs no human interpretation.
· • Repeatable: Test can be run many times in a row without human intervention.
· • Robust: Test produces same result now and forever. Tests are not affected by changes in the external environment.
· • Sufficient: Tests verify all the requirements of the software being tested.
· • Necessary: Everything in each test contributes to the specification of desired behavior.
· • Clear: Every statement is easy to understand.
· • Efficient: Tests run in a reasonable amount of time.
· • Specific: Each test failure points to a specific piece of broken functionality; unit test failures provide "defect triangulation".
· • Independent: Each test can be run by itself or in a suite with an arbitrary set of other tests in any order.
· • Maintainable: Tests should be easy to understand and modify and extend.
· • Traceable: To and from the code it tests and to and from the requirements.
Though the automation testing has many advantages, it has its own disadvantages too. Some of the disadvantages are:
• Proficiency is required to write the automation test scripts.
• Debugging the test script is major issue. If any error is present in the test script, sometimes it may lead to deadly consequences.
• Test maintenance is costly in case of playback methods. Even though a minor change occurs in the GUI, the test script has to be rerecorded or replaced by a new test script.
• Maintenance of test data files is difficult, if the test script tests more screens.
Some of the above disadvantages often cause damage to the benefit gained from the automated scripts. Though the automation testing has pros and corns, it is adapted widely all over the world.
Introduction to CMM
Quality software should reasonably be bug-free, delivered on time and within budget. It should meet the given requirements and/or expectations, and should be maintainable.
In order to produce error free and high quality software certain standards need to be followed.
ISO 9001: 2000 is Quality Management System Certification. To achieve this, an organization must satisfy ISO 9001: 2000 clauses (clauses 1 - 8).
Six Sigma is a process improvement methodology focused on reduction in variation of the processes around the mean. Its objective is to make the process defect free.
SEI CMM is a defacto standard for assessing and improving processes related to software development, developed by the software community in 1986 with leadership from SEI. It’s a software specific process maturity model. It provides guidance for measuring software process maturity and helps process improvement programs.
· Initial
· Repeatable
· Defined
· Manageable
· Optimizing
The software process is characterized as ad hoc, and occasionally even chaotic. Few processes are defined, and success depends on individual effort and heroics.
Basic project management processes are established to track cost, schedule, and functionality. The necessary process discipline is in place to repeat earlier successes on projects with similar applications.
The software process for both management and engineering activities is documented, standardized, and integrated into a standard software process for the organization. All projects use an approved, tailored version of the organization's standard software process for developing and maintaining software.
Detailed measures of the software process and product quality are collected. Both the software process and products are quantitatively understood and controlled.
Continuous process improvement is enabled by quantitative feedback from the process and from piloting innovative ideas and technologies.
| Features | CMM | ISO 9000 |
| Assess process maturity | Yes | No |
| Emphasizes continuous improvement | Yes | No |
| Provides self-assessments | Yes | Yes |
| Applicable to projects | Software only | No |
| Applicable to programs | Software only | Yes |
| Applicable to organizations (operations) | Software only | Yes |
| Requires a quality management system | Yes | Yes |
CMM ensures that the process followed for developing a product produces error free product. A company which is process driven is more successful than the company which is people driven. Hence a company needs to have a good process for software development for it to be successful.
Audit: An independent examination of a work product or set of work products to assess compliance with specifications, standards, contractual agreements, or other criteria.
Acceptance testing: Testing conducted to determine whether or not a system satisfies its acceptance criteria and to enable the customer to determine whether or not to accept the system.
Alpha Testing: Acceptance testing performed by the customer in a controlled environment at the developer's site. The software is used by the customer in a setting approximating the target environment with the developer observing and recording errors and usage problems.
Assertion Testing: A dynamic analysis technique which inserts assertions about the relationship between program variables into the program code. The truth of the assertions is determined as the program executes.
Boundary Value: (1) A data value that corresponds to a minimum or maximum input, internal, or output value specified for a system or component. (2) A value which lies at, or just inside or just outside a specified range of valid input and output values.
Boundary Value Analysis: A selection technique in which test data are chosen to lie along "boundaries" of the input domain [or output range] classes, data structures, procedure parameters, etc. Choices often include maximum, minimum, and trivial values or parameters.
Branch Coverage: A test coverage criteria which requires that for each decision point each possible branch be executed at least once.
Bug: A fault in a program which causes the program to perform in an unintended or unanticipated manner.
Beta Testing: Acceptance testing performed by the customer in a live application of the software, at one or more end user sites, in an environment not controlled by the developer.
Boundary Value Testing: A testing technique using input values at, just below, and just above, the defined limits of an input domain; and with input values causing outputs to be at, just below, and just above, the defined limits of an output domain.
Branch Testing: Testing technique to satisfy coverage criteria which require that for each decision point, each possible branch [outcome] is executed at least once. Contrast with testing, path; testing, statement. See: branch coverage.
Compatibility Testing: The process of determining the ability of two or more systems to exchange information. In a situation where the developed software replaces an already working program, an investigation should be conducted to assess possible comparability problems between the new software and other programs or systems.
Cause Effect Graph: A Boolean graph linking causes and effects. The graph is actually a digital-logic circuit (a combinatorial logic network) using a simpler notation than standard electronics notation.
Cause Effect Graphing: This is a Test data selection technique. The input and output domains are partitioned into classes and analysis is performed to determine which input classes cause which effect. A minimal set of inputs is chosen which will cover the entire effect set. It is a systematic method of generating test cases representing combinations of conditions.
Code Inspection: A manual [formal] testing [error detection] technique where the programmer reads source code, statement by statement, to a group who ask questions analyzing the program logic, analyzing the code with respect to a checklist of historically common programming errors, and analyzing its compliance with coding standards.
Code Review: A meeting at which software code is presented to project personnel, managers, users, customers, or other interested parties for comment or approval.
Code Walkthrough: A manual testing [error detection] technique where program [source code] logic [structure] is traced manually [mentally] by a group with a small set of test cases, while the state of program variables is manually monitored, to analyze the programmer's logic and assumptions.
Coverage Analysis: Determining and assessing measures associated with the invocation of program structural elements to determine the adequacy of a test run. Coverage analysis is useful when attempting to execute each statement, branch, path, or iterative structure in a program.
Crash: The sudden and complete failure of a computer system or component.
Criticality: The degree of impact that a requirement, module, error, fault, failure, or other item has on the development or operation of a system.
Cyclomatic Complexity: The number of independent paths through a program. The cyclomatic complexity of a program is equivalent to the number of decision statements plus 1.
Error: A discrepancy between a computed, observed, or measured value or condition and the true, specified, or theoretically correct value or condition.
Error Guessing: This is a Test data selection technique. The selection criterion is to pick values that seem likely to cause errors.
Error Seeding: The process of intentionally adding known faults to those already in a computer program for the purpose of monitoring the rate of detection and removal, and estimating the number of faults remaining in the program. Contrast with mutation analysis.
Exception: An event that causes suspension of normal program execution. Types include addressing exception, data exception, operation exception, overflow exception, protection exception, and underflow exception.
Exhaustive Testing: Executing the program with all possible combinations of values for program variables. This type of testing is feasible only for small, simple programs.
Failure: The inability of a system or component to perform its required functions within specified performance requirements.
Fault: An incorrect step, process, or data definition in a computer program which causes the program to perform in an unintended or unanticipated manner.
Functional Testing: Testing that ignores the internal mechanism or structure of a system or component and focuses on the outputs generated in response to selected inputs and execution conditions. (2) Testing conducted to evaluate the compliance of a system or component with specified functional requirements and corresponding predicted results.
Integration Testing: An orderly progression of testing in which software elements, hardware elements, or both are combined and tested, to evaluate their interactions, until the entire system has been integrated.
Usability Testing: Tests designed to evaluate the machine/user interface.
Validation: Establishing documented evidence which provides a high degree of assurance that a specific process will consistently produce a product meeting its predetermined specifications and quality attributes.
Validation, Verification and Testing: Used as an entity to define a procedure of review, analysis, and testing throughout the software life cycle to discover errors, determine functionality, and ensure the production of quality software.
Volume Testing: Testing designed to challenge a system's ability to manage the maximum amount of data over a period of time. This type of testing also evaluates a system's ability to handle overload situations in an orderly fashion
