Why is Independent Software Testing Critical?

The Story

It was about a decade ago, and the Northern California sun shone in its full glory as I crossed the Bay Bridge enroute to the East Bay town of Concord. I was about to take on the largest testing programme I’d ever managed: the global implementation of SAP’s full suite of applications for one of the largest petrochemical concerns on the planet. Although based in San Francisco, the $500m project employed 1,000 people in California, Manila, Rio, and Cape Town. The task was daunting: with a team of 225 people scattered around the world, ensure that this system would run as intended and not impede the USD150b company’s ability to function. Failure would be professionally fatal. 

I remember my meeting with the General Manager of the programme, a slightly frail, wispy man with a huge heart and an even bigger mind. He was crystal clear on what he expected from me, and I will never forget his words: “Rule with an iron fist”, was his guidance. These were not the words I would ever have expected to emanate from the mouth of an aging Northern California hippie. But his objectives were abundantly transparent. The huge team tasked with designing a massive, unimaginably complex system had a very finite budget of time and currency. Despite this, the standard process of running through 3 cycles of System Integration Test was extended to 5. This programme would have quality at the forefront of its philosophy, and nothing would compromise this. As the System Readiness Lead, I was given an independent reporting line directly to the General Manager and was directed to brief him, the CIO, and the programme leaders on the health of the quality effort without mute or a filter. 

The reason for the mandate should be obvious. A project organisation of 1,000 people possesses endless opportunities to veil the real state of quality. The belief was that only by positioning a neutral and independent party in the thick of things could those ultimately accountable for success feel certain they had their finger on the pulse of the project. 

Testing is, by design, in a precarious position for any software project by virtue of its location at the end of the delivery lifecycle. The concept of ‘squeeze’ will be familiar to anyone who has ever run a testing programme. Its most typical meaning is a reduction in the time available to the test team to get their work done because of overruns upstream in the delivery cycle. This situation places the owner of testing at a very precarious decision point: where do we cut corners to make our cost and time budget? Or do I stand my ground and rule with an iron fist? These questions are best answered in the context of testing independence. 

The Why 

The answer to the question of ‘why’ test independence is critical should now be abundantly clear. In a situation where one organisation running the full project, like a systems integrator, for instance, has allowed upstream activities like design or building to run overbudget, the owner of that budget must either absorb those overruns by compromising on the last phase of the project, i.e. testing, or go ‘back to the well’ and ask for more money, time, or both. In a situation where testing was run by an independent entity with their own budget, this could not happen. But aside from the budget, there are myriad reasons why a segregation of duties makes sense.    

Bias – Testing one’s own code 

Developers are people too, right? And like all people, they have feelings and egos that can be bruised. And while no one would contend that testing is not a part of a developer’s daily routine, once outside of unit testing and as part of a larger system, the validation of code should become the domain of an unbiased party. 

Resources dependencies and constraints  

Finances are finite in our world, so unless your budget is a blank cheque, you will have choices to make. Hiring a permanent staff for a temporary project simply makes no economic sense. This is where engaging a third party for the lifetime of your project not only brings unbiased quality, it also enables costs budgeted for a project to be time limited. Testing service providers offer the right skills when you need them and, especially when using an offshore force, at a cost that will please your CFO. 

Acceleration  

Being able to ramp up quickly and utilise the knowledge that has been built through front-line experience across the industry spectrum is why consulting as a business model exists. Where an in-house team or an external development partner may be accessible, tapping into the specialist skills of an organisation solely focused on independently validating the quality of the solution you are implementing makes good business sense. Partners like this bring methodology, tools, and artefacts like automated test scripts to the table, as well as hard to find niche expertise like performance and security testing, data validation, cutover, disaster recovery, cloud migration, and a slew of other capabilities that are extremely difficult to maintain in-house. 

Conflict of interest 

You may have a great systems integrator. Speaking as a former SI, I can attest to the fact that there are many consultants out there with exemplary ethics. The fact remains that a programme manager or development lead is judged on his or her ability to manage a project to budget. It is here that, shall we say, an “obfuscation of the facts” may occur. A hesitancy to reveal issues and cut corners to preserve margins is tempting in the closing phases of a large IT project. Removing this temptation by eliminating the conflict of interest and segregating responsibilities is the right thing to do. 

Cost 

The implications for the budget are multifaceted. The efficiencies associated with using a quality assurance specialist have already been discussed. Engaging a team based in lower-cost regions of the world further stretches your currency. But one critical cost component that often flies under the radar is the opportunity cost of using your business analysts or developers for testing work. One theme has come up often during conversations with clients about using your own people for work like testing: They are not trained for it, they don’t particularly enjoy doing it, and diverting them to a project means that other valuable work may suffer.  

The How 

The benefits of testing independence resemble a bell curve plotted along the timeline of the waterfall SDLC. In the development phase of the project, developers will unit test their own code. Once we move to system testing, independent testers can be phased in as the system becomes increasingly stable. Once system integration testing begins, function-driven testing and the defects that this throws off (in addition to the manpower required) is where independent testing pays dividends in efficiency and honesty. The objective of this phase is not only to ensure that the integrated system holds together, but also to make sure that your business users tasked with User Acceptance Testing do not throw their computers out of the window in protest at a system that is not ready for prime time. It is here that an independent testing team may be used to augment your business team and help facilitate the test itself. 

Models for Enterprise SW testing 

There is no titanium-clad methodology to ensure you can realise the benefits of test independence in your organisation. The main facet of the approach is to stick to a ‘segregation of duties’ rule. Think of an automobile assembly line: one worker installs the drivetrain into the chassis, and the unit moves on to a totally separate group to validate that the bolts are torqued. Some derivatives can include the following: 

• 100% In House – if you have the workforce and a consistent flow of demand, a distinct QA organisation can be formed and maintained in your company. The reality, however, is that demand is almost never constant, and the demand peaks will necessitate some outside support. 
• 100% Sourced to 3rd Party Application Development Partner / SI – The other side of the spectrum is that the QA function is trusted to your application developer or systems integration partner. The advantage of this common model is that it’s easy: one partner, one ‘throat to choke’, as they say. This, of course, opens the door to a manipulation of the facts. 
• In-house testers augmented by a 3rd party Testing Services Provider or, in reverse, by 3rd party Testing Service Provider augmented by your BAs – these approaches only differ in who is running the testing programme. Ultimately, they provide the same benefits: test independence and the ability to scale and contract to meet demand. 
• 100% sourced to 3rd Party Testing Service Provider – The Managed Test Services model has several benefits. First and foremost, it ensures that your BA’s and developers can focus on what they were hired to do: the design and development of software. It also creates a relationship with a partner that can mould itself according to your needs. 

Conclusion 

We, fortunately, do not live in a world where the people who assembled the Boeing 787 you may be flying in are also the people who certify that the aircraft is airworthy. A complex system of processes, tools, and people is deployed to ensure an independent view of quality is achieved. Lives may not depend on the quality of your software (although in many domains, they certainly do), but the success of your business may well hang in the balance. Just as any large corporation hires an independent auditor to verify its financial reporting, the verification that your enterprise software is ready for the real world should be left to your independent testing provider. Your iron fist. 

As the world’s leading Independent Quality Engineering & Software Testing services company, Cigniti brings the power of AI into Agile and DevOps to accelerate enterprise digital transformation. We assist global organizations in accelerating the time to market by predicting and preventing unanticipated failures by leveraging AI-driven, proprietary Continuous Testing and Test Automation solutions that are platform and tool agnostic, with customer centricity at the core of the transformation.  

To join the bandwagon of these successful organizations, schedule a discussion with our software testing experts to understand more about why independent software testing is critical.