Measurement and Metrics for Test Management

Lord Kelvin said “To measure is to know”. Measuring something is a very critical activity because “If you cannot measure it, you cannot improve it”. So very apt for test measurement and metrics!

Metric, in common parlance, refers to “a system or standard of measurement.” For example, the mileage a car actually gives per litre of fuel compared to the mileage claimed by the manufacturer.

In terms of software, testing metrics (also known as test measurement) is the “quantitative measure of the extent, capacity, dimension, amount or size of some attribute of a system, system component, or process.”

Simply put, metrics is the quantitative measure using which we can estimate and verify the progress, quality, and health of a software testing effort. For example, the total number of defects present in an application.

Software testing metrics are used to analyse a software testing process in terms of performance, functionality, etc., and then using the data to improve its the quality and efficiency.

The Need for Measuring Test Metrics

Measuring Test Metrics is a highly business-critical activity that helps us:

• Understand the specific type and amount of improvement required
• Make correct and proper decisions related to technology or process change
• Take well-informed decisions for the corrective actions to be taken
• Gather fool-proof evidence of the claims made

Things to Consider While Identifying the Metrics

While collection and measurement of metrics is important, it is even more important to understand and define the correct testing metrics. If not done properly and with due diligence, collection of wrong data will have a double negative impact – not only would you have collected data that is of no use, you would also have wasted a lot of time and efforts on the same.

Here’s a list of a few things that must be taken into consideration while specifying test metrics:

• Identify the target audience that needs the specific data – who are you collecting the data for?
• Define a well-established goal and purpose for the metric being collected – and share it with the entire team
• Explain all relevant and related metrics that are required for the project in question
• Spend time to understand the cost impact or benefits of spending efforts on the collection of each identified metric, and allocate the phase that will benefit the most

Types of Metrics

Metrics are usually of the following three types:

• Product related metrics: These metrics relate to the quality of software products and identifying them and taking corrective actions helps improve the quality of the product.
• Process related metrics: These metrics, if identified and analyzed properly, help improve the efficiency of a process, for example, SDLC.
• Project related metrics: These metrics are used to measure the efficiency with which a project is being run, and the tools being used in the project.

Different Stages of the Testing Metrics Life cycle

The Testing Metrics Life cycle usually consists of 4 phases: Analysis, Communication, Evaluation, and Report generation. Following are the brief details of the specific activities that need to be performed in each phase:

• Analysis
  • Identify the specific metric that is to be measured. For example, the process related to project tracker.
  • Keeping in mind the identified metric as the base, define clearly the metrics as required. For example, the number of test scripts to be executed per day.
• Communication
  • Explain the need for metric to stakeholder and testing team: This will help the teams understand the importance of collecting details for the specific metric.
  • Educate the testing team about the data points need to be captured for processing the metric: Identify and explain to the team the specific details that need to be tracked, determine the tracking frequency, and allocate the responsible resource. For example, at the end of each working day, the Manager of the Testing Team will generate a collated report of the 200 test scripts that are executed daily.
• Report generation
  • Develop the report with effective conclusion: Identify the improvement areas based on the interpretation and analyses of the defined metrics. For example, if the number of test cases executed are less than the goal of 200, you need to conduct a proper investigation to understand the causes, and then take the corrective actions – whether to reduce the number of tests to be executed, or rectify what was causing the fall in numbers.
  • Distribute report to the stakeholder and take feedback.

At Cigniti, we cover all the bases and ensure that correct metrics are collected for executing the right set of tests. We ensure the best quality for your product and that your customers are happy. Our tool agnostic test automation frameworks ensure accelerated testing so that you get higher productivity and an enviable time to market.