Calibration – that's what you need

For anyone who has ever had to follow a process to produce something, it quickly becomes obvious that measurement and testing are important to achieve an acceptable result.

Whether making scrambled eggs or building a space vehicle, measuring everything – from the quantity of eggs, butter and salt, to rocket components, assemblies and fuel – is important. Making those measurements accurately is even more important, or you risk annoying the person eating the eggs or wasting millions of dollars on a failed space mission.

We go to some lengths, then, to ensure measurements are accurate, including the methods and equipment used to perform the measurements and testing. We often call this ‘calibration’.

Since internal auditing is also a process, shouldn’t it then follow that there ought to be some kind of checking to ensure that audit results are accurate? After all, management needs to be confident in those results to make better decisions about the implementation of the quality management system.

Should an organisation, therefore, perform some type of ‘calibration’ on the internal auditors – since it is they who perform the measurement/testing on the quality management system? Of course, although it may seem odd to compare an auditor to a ‘standard’ and to contemplate measurement ‘uncertainties’, these are all applicable, in some way.

Auditor training

Let’s start where most internal auditors start, which is by attending a training course. Thinking about the relationship to ‘calibration’, what is an auditor taught that can be used as a ‘standard’ against which their results can be evaluated?

1. The ‘standard’ used is ISO 9001 (or an equivalent) and;
2. The audit process is based on external auditing (typically replicating the conformity assessment body (CAB) or supplier audit process)

Although there may be similarities between the internal and external audit process, there are key points where the two can diverge significantly.

This can be likened to understanding if each activity in the audit process is linear, as shown here:

We can see there’s quite a bit of difference between the two audit processes, most notably in the time allocation and, specifically, in planning. With access to much more data and information for the internal auditor, there’s greater flexibility in selecting audit scopes and criteria. Hence, the audit durations can be commensurate with those various scopes and criteria. This, in turn, leads to less complex reporting.

The variety of available scopes and criteria also presents challenges with ensuring auditors are competent. An assignment to audit a product design project is going to be more complex than, say, a product mechanical assembly. Therefore, this complexity should factor in the definition, and related evaluation, of auditors’ competencies.

"Since internal auditing is also a process, shouldn’t it then follow that there ought be some kind of checking to ensure audit results are accurate?"

Andy Nichols CQP FCQI, business advisor

Contrast the relative amounts of time spent and we can see that because of (primarily) commercial and accreditation constraints on the audits, the time spent by external auditors is very different. Typically, the scope and criteria are pretty much fixed as being the whole quality management system and ISO 9001, respectively.

Most available auditor training is centred on the performance of a single audit using this ‘fixed’ audit scope and criteria, which is not applicable after certification. Reporting from external audits involves a much wider audience – from the audit client to the CAB management, to the CAB’s accreditation body, as well.

Auditor objectivity and bias

Having trained and gained some experience in auditing, what then? In measurement terms, shouldn’t the organisation understand if the auditor(s) develop a bias? We study for that when performing physical measurement, so why not when auditing?

There are two potential influences at play here: 

• The four stages of competence – even experienced auditors can lose competency, usually when things change.
• The Ladder of Inference – developed by Harvard professor Chris Argyris, this affects how people interpret what they observe and perceive.

Both of these influences manifest themselves over time and can lead to less-than-optimal results. Since both are linked to auditor experience over time, they might be considered as being linked to developing ‘bias’ or even ‘stability’ of the audit results.

Calibration method

Since we are dealing with people, we cannot draw a strict comparison to the calibration or verification of an item of test equipment. However, one method that may have been experienced, particularly in the domain of third-party (CAB) auditing is ‘shadowing’. This is where an auditor from the CAB (often management) observes the assigned auditor performing at their client.

It may also be an accreditation body ‘witness audit’ that is being performed. This might be considered as verification against a standard – set by the observing auditor. This approach may be subject to what’s known as the ‘Hawthorne Effect’ – similar to what we see happen on roads when a police car is present, and everyone slows their vehicle – it isn’t always an effective verification technique. Auditors being observed may adjust their techniques/behaviours because they know they’re being watched.

Since ISO 19011, the guidance for management systems audits references the use of ‘testing’, so it might be appropriate periodically to set verification tests. For example, providing the auditors with synthesised audit scenarios and discussing the situations described can provide insights into what an auditor would do when faced with similar circumstances. This can be applied to the review of (previously) reported nonconformities, including relevant auditors’ notes, to see if there’s agreement on the findings, after the passage of time.

Often overlooked as a suitable verification method is the use of checklists. Although audit checklists are often the subject of much debate, their creation has useful benefits – both in addressing Ladder of Inference issues (described above), as well as those discovered by Nobel Prize winner Richard Fynmann for learning any subject quickly.

In conclusion then, while we know that it’s rarely effective, when things don’t go as planned to retrain the operator as a corrective action, there’s certainly an opportunity to use the analogy of calibration to ensure that auditors don’t ‘stray’ when performing their assignments. Now, where shall we put the ‘Calibrated’ stickers…