Return on Investment for Automation

With the rapid advancement of technology, software test automation has assumed a crucial role and become indispensable in the software industry. Due to time constraints, automated tests have taken precedence over manual tests. However, it’s important not to dismiss manual testing entirely, as not all tests can be automated. The key point I want to emphasize is that the efforts of manual testers can be effectively utilized for other purposes, even when automated scripts are running. In essence, automate some or all tests and allocate manual testing efforts more efficiently to other tasks.

Return on Investment (ROI) doesn’t necessarily have to translate into financial benefits or cost savings. Many individuals make mistakes when calculating the ROI for test automation. The ROI for automation testing doesn’t solely involve calculating the time and effort required to work on tests, considering they still need manual execution. This aspect is often overlooked by many effort estimators or organizations, and it’s crucial for them to understand that manual effort should be factored into the consideration of automation scripts.

I have outlined a couple of scenarios explaining why automation is necessary and the benefits it brings to the organization. It’s important to note while reading this post that it reflects my opinion based on experiences gained. I welcome valuable feedback in response to this post.

Scenario-1 – Cross browser testing

In a customer project, the task at hand is to conduct Cross Browser testing for 100 predetermined functional tests. The test team scrutinized the requirements for automation and posed fundamental questions for the process:

1. Which web browsers does the application require testing on? (e.g., Chrome, Microsoft Edge, Safari, Firefox, etc.)
2. What are the versions of each web browser that the application needs testing on?

Following client approval, the testing team opted to distribute these 100 functional test cases across multiple browsers. However, it is crucial to assess the risks already acknowledged by the customer. The planned testing approach may potentially limit the opportunity to uncover anomalies, as only a specific subset of test cases will be executed for each browser. This compromise could result in a certain degree of quality compromise. Browser vulnerabilities are commonly observed in numerous applications.

Possible solution:

In such scenario, if the solution is automated all 100 test cases for each browser, even though the total number is increased by 4X, all the test cases are covered for all browsers and the quality is not compromised.

Alternative Solution:

Prioritise the test cases based upon criticality of browser and functionality and split the test cases accordingly. However, this still compromises the quality.

Benefits of Automation of this approach

1. Number of test cases increased so, more coverage, in this case, 100%
2. Quality not compromised, i.e. testing quality improved by testing all test cases on all browsers
3. Manual testers can be utilised for other testing work – meaning more productivity and efficiency.
4. Automation can be used for retesting, which could save time. i.e. these cases be run in multiple cycles

Here, the RoI is not purely financial benefits but it improved the coverage and also testing efforts can be utilised efficiently.

Scenario-2 – Functional Regression

Problem:

The project testing team decides to write 250 Regression test cases for first release for certain number of scenarios and decided to write test cases. However, desirable or low priority requirements were discarded due to time constraint to the schedule and/or test resource constraint. Finally the test team arrived at a decision to write 150-175 test cases, after client team’s approval.  Team of 3 testers were deployed to execute 150-175 test cases with reduced effort and leaving other non-priority test cases. Remaining 75-100 test cases were not covered, which would take additional 3-4 days approximately.

In such scenario, the solution can be automated, where the execution time can be saved and utilised this time to execute uncovered test cases.

Though the RoI has no financial benefits to the extent, the above scenario has improved the quality of the solution or application that has been tested.

Benefits, if it would have been automated

1. Increased the coverage by 30%
2.  using resources efficiently

Conclusion:

Automation improves the quality and saves time over the time, though setup costs are high. Redirection of manpower can be the best opportunity for organisations to save money.

Note: This article didn’t articulate how to calculate RoI but explained how RoI is not just returns in terms of money and utilise the resources efficiently.