The Effectiveness of QA Automation and Manual Testing

Nowadays, more and more attention is paid to software testing as an important component of QA. Very often there is an opinion that with high-quality code development by highly qualified specialists, testing is either not necessary at all, or is only minimally needed. 

But in real life, everything happens a little differently. Even very well-written code can work separately, but when integrated with other modules, errors of various kinds can occur. Or, the software behaves differently when porting to a new platform.

Therefore, it is necessary to pay attention to various aspects of QA, including both manual and automated software testing. Let's consider what types of testing are available and how they can effectively help information systems operate reliably and without failure.

The Importance of QA in Software Development

Quality Assurance (QA) is a way of preventing errors and defects in manufactured products and preventing problems in delivering products or services to customers. ISO 9000 defines it as “the part of quality management aimed at providing confidence that quality requirements will be met”.

The terms quality assurance and quality control are often used interchangeably to refer to ways of ensuring the quality of a service or product. For example, the term “assurance” is often used to describe the implementation of inspection and structured testing as a quality assurance measure in a product or information system software design. However, the term control is used to describe the fifth phase of the Define, Measure, Analyze, Improve, Control (DMAIC) model. DMAIC is a data-driven quality strategy used to improve processes.

There are selective, solid, and statistical types of QA. All products undergo complete control. In production, records of all defects in the process of manufacturing the product are kept.

Selective is control of a part of the product, the results of which are valid for the entire batch. This type is precautionary, from here it is carried out throughout the production process in order to prevent the occurrence of defects.

Incoming control is checking the quality of raw materials and auxiliary materials entering production. Constant analysis of the supplied raw materials and other materials allows the user to influence the production of supplier enterprises, seeking to improve quality.

Interoperational control covers the entire technological process. This type is sometimes called technological, or current. The purpose of interoperational control is to check compliance with technological regimes, storage, and packaging rules for products between operations.

Output (acceptance) control - quality control of finished products. The purpose of the final inspection is to establish the conformity of the quality of finished products to the requirements of standards or technical specifications, to identify possible defects. If all conditions are met, then the delivery of products is permitted. The Quality Control Department also checks the quality of packaging and the correctness of the labeling of finished products.

As mentioned, “quality assurance” includes two principles: "Fit for purpose" (product must fit for purpose); and "right the first time" (errors must be corrected). QA includes quality management of raw materials, assemblies, products and components, production-related services and management, production processes, and inspection. Two principles are also evident in the development of a new technical product: the task of engineering is to make it work once, and the task of quality assurance is to make it work consistently and reliably.

We can define a process of QA in many ways, for instance as:

• from a subjective user-based approach, which contains "various weights that people usually associate with quality characteristics",
• a value-based approach that finds consumers who associate quality with price and draws general conclusions about quality based on such relationships.

Software Testing

Software Testing is a technical research process designed to identify information about the quality of a product in relation to the context in which it is to be used. Testing techniques also include the process of finding errors or other defects, as well as testing software components for evaluation.

Software testing is the process of verifying compliance with the stated requirements for the product and the actually implemented functionality.  It is carried out by observing its work in artificially created situations and conducting a limited set of tests selected in a certain way.

Software testing can be evaluated using the following criteria:

• compliance with the requirements guided by designers and developers
• the correct answer for all possible inputs
• performing functions in a reasonable time
• practicality for the selected software or system
• compatibility with software and operating systems
• compliance with customer objectives.

Manual Testing

Manual testing is part of the quality control testing process in the software development process. It is performed by a tester without using software tools to test a program or a website by simulating user actions. Ordinary users can also act as testers, informing developers about the bugs found.

Manual testing can reduce errors, identify and eliminate bottlenecks, ensure stable system performance, evaluate usability, and, as a result, produce a better product that meets user expectations.

There are two main types of manual testing: black-box testing and white-box testing.

Black Box testing is used when testers do not know how the system under test works. Black-box testing, also known as specification-based testing or behavior testing, is a testing technique that relies solely on the external interfaces of the system under test.

According to ISTQB, black box testing is:

• testing, both functional and non-functional, not involving knowledge of the internal structure of a component or system.
• test design based on the black box technique - a procedure for writing or selecting test cases based on an analysis of the functional or non-functional specification of a component or system without knowing its internal structure.

In this case, the program under test for the tester is like a black opaque box, the contents of which he does not see.

The tester has no idea about the structure and internal structure of the system and concentrates on what the program does, not how it does it.

White-box testing is used when the tester knows all the details of the implementation of the program under test. White-box testing (also: transparent, open, glass box; code-based or structured testing) is a software testing method that assumes that the internal structure/design/implementation of a system is known to the tester. The input values are selected based on knowledge of the code that will process them. In the same way, the tester knows what the result of this processing should be. Knowledge of all the features of the program under test and its implementation is mandatory for this technique. White-box testing looks deeper into the internal structure of the system, beyond its external interfaces.

According to ISTQB, white-box testing is:

• testing based on an analysis of the internal structure of a component or system.
• test design based on the white box technique - a procedure for writing or selecting test cases based on an analysis of the internal structure of a system or component.

The program under test for the tester is a transparent "white" box, the contents of which he sees perfectly.

Main advantages of manual testing:

• a manual test project can be launched quickly
• manual testing allows you to make adjustments (change and update tests) if necessary
• easily adapts to dynamic changes in the system under test
• manual testing fully simulates the actual use of the system by the end-user
• allows you to get user feedback, especially on one-time and short-term projects

Automation Testing

Automated software testing is part of the QA testing process in the software development process. It uses software tools to execute tests and verify the results of the execution, which helps reduce testing time and simplify testing.

The first mention of automated testing was in the book by Frederick Brooks "The Mythical Man-Month", which talks about the prospects for using unit testing, but it wasn't until the 1980s that test automation really took off.

There are two main approaches to test automation: code-level testing and user interface testing (specifically, GUI testing). The first type is unit testing and the second is the imitation of user actions, functional testing (using special test frameworks).

Effectiveness of Automation Testing and Manual Testing

The effort and effectiveness of automated and manual testing are measured in terms of total test time, which includes script generation and test execution for automated testing, and includes test execution and reporting for manual testing. 

Efficiency is measured by the number and severity of detected defects, as well as by the amount of time spent on identifying a defect. since, very often, corrections in the functioning of an information system are critical and require immediate correction because end-users cannot work normally. Efficiency results often show that automated testing requires a lot of initial effort, mainly caused by scripting. On the other hand, these costs can be recouped over time, since automated tests are performed a large number of times for regression testing. Very often, automated testing is more effective in detecting defects than manual testing.

Modular testing plays an important role in automating the software testing process. Unit testing is a software testing method that consists of testing each module of the program code separately. The module is the smallest part of the program that can be tested. In procedural programming, a module is considered a separate function or procedure. In object-oriented programming, this is a method.

Modular tests, or unit tests, are created during development by programmers and, sometimes, white-box testers. Unit tests are usually used to make sure that the code meets the requirements of the architecture and has the expected behavior.

The purpose of modular testing is the isolation of each part of the program and the certainty that each individual part is correct. The modular test provides a rigid "contract" under which the tested code should work. As a result, it provides some benefits. Modular testing helps to find errors earlier in the software development cycle, which makes development cheaper and faster.

The most common form of automation is testing applications through a graphical user interface (GUI). The popularity of this type of testing is due to two factors: first, the application is tested in the same way that a person will use it, and second, it can be tested without having access to the source code.

GUI test automation has evolved over many generations of tools and techniques:

• The capture/playback tools record the tester's actions during manual testing. They allow tests to be performed without direct human intervention for extended periods of time, significantly increasing productivity and eliminating dull repetition of repetitive tasks during manual testing. At the same time, any small change in the software under test requires rewriting the manual tests. Therefore, this first generation of tools is neither efficient nor scalable.
• Scripting is a form of programming in languages specifically designed for automating software testing — alleviates many of the problems of recording and playback tools. But the development is done by high-level programmers who work separately from the testers who run tests directly. In addition, scripts are most suitable for testing the GUI and cannot be embedded, batched, or in any way combined into a system. Finally, changes to the software under test require complex changes to the corresponding scripts, and maintaining an ever-growing library of test scripts becomes ultimately an insurmountable task.
• Data-driven testing is a methodology used in test automation. A special feature is that test scripts are executed and verified based on data stored in a central data warehouse or database. Data-driven testing is the combination of multiple interacting test scripts and their data sources into a framework used in a methodology. In this framework, variables are used for both input values and output check values: in a test script, navigation through the application, reading data sources, and maintaining test logs are usually coded. Thus, the logic to be executed in the script also depends on the data.

Let's list the methods to make manual software testing more efficient:

• learn to create correct and effective documentation (test cases, test scenarios, or software specification requirements).
• learn to pass as many test cases (positive and negative) as possible.
• concentrate on both the functionality of the application and its appearance.
• get in the habit of taking a screenshot for every bug or problem.

To increase the efficiency of both manual and automated testing, it is worth turning to system tools and standards, for example, the main ones:

• Quality Management Systems (QMS)
• auditing and procedural documentation writing CMMI
• Six Sigma
• Measurement Systems Analysis (MSA)
• Quality Function Deployment (QFD)
• Failure Mode and Effects Analysis (FMEA)
• Advanced Product Quality Planning (APQP)

If you look at the problem of efficiency from a practical point of view, then in real projects the most important aspect is the choice of which part of testing to transfer to the manual testing department, and which part needs to be provided by automated testing.

It is also very important to determine the moment when a certain stage of the project allows testing to begin. For example, in V-model, testing can be carried out at all stages of development, and this allows you to correctly build the process and make testing very effective. Also, in the agile model, each iteration ends with testing and delivery of a working version of the software, as it is perfectly proven in the Scrum methodology.

Conclusion

One of the main problems of automated testing is its labor intensity: despite the fact that it allows the elimination of some of the routine operations and speeds up the execution of tests, large resources can be spent on updating the tests themselves. This applies to both types of automation. When refactoring, it is often necessary to update unit tests as well, and changing test code can take as long as changing the main code. On the other hand, when the application interface changes, it is necessary to rewrite all scripts that are associated with the updated windows or web pages. It can take a large number of tests, and use up significant resources.

Our specialists from Svitla Systems very clearly understand and correctly carry out projects, including in terms of the QA process. Our project managers and qualified test engineers create highly effective test plans and balance the balance between manual and automated testing. We carry out testing projects for various software development technologies, including web, desktop, and mobile applications. Our company forms development teams and effectively calculates the number and scope of testing required for each specific project. You can contact Svitla Systems for both manual testing services and automated testing as well as the development of modern software systems.