7 Questions to Benchmark Your Quality Assurance and Testing Practices
The frequency of enterprise software releases and adoption of cloud and mobile technologies are just some of the factors driving increased spend and focus on quality assurance and testing.
But QAT practices can vary greatly from one IT organization to the next. So how do you stack up against others? This short assessment should help you benchmark your quality initiatives.
The data I cite below comes from the 2014-15 World Quality Report, a comprehensive survey of over 1,500 global IT leaders and testing professionals published by HP, Capgemini and Sogeti. I’ve had the good fortune of participating in analysis for the report over the past year as a QA and Testing Evangelist with HP Software.
1) What percentage of your organization’s overall IT budget is devoted to application Quality Assurance and Testing (QAT)?
IT organizations, on average, devoted 26% of their 2014 budget to QAT, up from 23% in 2013 and 18% in 2012. In years past, the majority of that spend has been devoted to hardware, but in 2014, the largest share of dollars are allocated to personnel (35%), followed closely by hardware and infrastructure (33%) and testing tools and licenses (32%). Granted, there is no magic percentage spend that should go towards quality. It depends on the size, maturity and needs of your organization.
2) Does your organization have a fully-operational Test Center of Excellence (TCOE)?
About 26% of IT organizations say they have a fully operational Test Center of Excellence in-house, up from 19% the previous year and just 4% in 2011. Another 54% of IT organizations say they are building or plan to build a TCOE either in-house or through outsourcing.
3) At what phase of the application delivery lifecycle do you engage your QA & T resources?
Just 35% of IT organizations involve their QA and test leads in initial planning and requirements to ensure quality throughout the delivery process. But 31% of organizations don't involve test resources until the implementation stage or later, relegating them to a reactive "find and repair" function. Of those who state they introduce testing late in the lifecycle, over two-thirds (68%) say they would like to introduce testing earlier to influence overall application quality.
4) Do you test mobile applications and devices utilized by your organization?
Nearly nine out of 10 IT organizations (87%) report that they have implemented new methods and tools for testing the security, performance and functionality of mobile applications and devices. That's up significantly from 55% in 2013 and 31% in 2012.
5) Do you utilize cloud-based testing?
Nearly a third (32%) of organizations say they rely on a cloud-based infrastructure for testing, up from 25% in 2013. This upward trend should continue for the foreseeable future because of the speed and cost advantages of building and maintaining test environments and sharing testing tools through the cloud.
6) Have you adapted your testing practices for agile development?
While 93% of IT organizations are using agile methodologies, nearly two-thirds (61%) say they lack a good testing approach that fits with agile development methods. In particular, 55% say they are challenged to apply test automation to agile projects. Because most testing methodologies and tools are based on traditional waterfall methods, IT organizations using agile development will need to adapt to new approaches and collaboration between testers and developers.
7) Does your organization utilize test automation?
Most testing organizations (89%) utilize some level of automation. On average, 28% of all test cases in these organizations are automated. By 2017, QA and testing managers say they would like automate as much as 40% of their test cases. A small minority of respondents rely heavily on automation. 7% of IT organizations say they have automated more than 70% of their test cases.
Want to learn more? If you are attending StarWest, I'll be part of this panel: World Quality Report 2014-15: Emerging Testing Trends on Wednesday, October 15, 2014 - 3:00pm - 4:00pm.
Sultan Syed, David Kapfhammer and I will be participating in a review of this year’s World Quality Report during this October 29, 2014 webinar.
Mike Cooper is VP of Quality Assurance and Testing for Healthcare IT Leaders. He previously led QAT teams and initiatives at HP Software, T-Mobile USA, and Equifax.
