In the previous article (Designing Subjective Performance Measures (Part 1)) we defined the typical performance levels used in subjective performance measures. In this article we will look at how to set these performance levels by defining the scoring process.

Defining the Scoring Process for a Subjective Performance Measure

Before starting it is important to use the general principles and approaches for setting the performance levels that have been previously provided (see Setting the Performance level (Part 1)). These apply equally for both objective and subjective performance measures.

However, in our experience there are two key aspects for describing subjective performance as follows:

1. attribute(s) that describes the performance; and
2. how often the attribute is observed (e.g. the frequency of occurrence).

For ease of use, we further highlight the relationship between these two aspects using a table with an example provided below in Table 1.

Subjective Performance Measure Attributes

The performance attributes for subjective performance measures are simply those characteristics that make up or describe the overall measure. For example, while many PBC practitioners want to measure single humanistic characteristics such as relationship and trust, the question is how to measure these in a repeatable way that assists both buyer and seller in the scoring process?

In this case, the performance measure attributes give a method where, used together with frequency (see below), we can define the attributes that make the overall qualitative characteristics. For example, in “The Trusted Advisor” by David Maister, Charles Green, Robert Galford, they define trust as a relationship between Credibility, Reliability, Intimacy and Self-orientation which each of these attributes having further definitions.

We have found that by using introductory words “The contractor . . .” together with a word picture for the attribute helps both buyer and seller understand the requirement. For example, “The contractor . . . Always . . . resolves disputes as the lowest possible level.”

Subjective Performance Measure Attribute Frequency

Linked to the performance measure attributes is how often the attribute is observed; that is the frequency of occurrence. While it is possible to use a quantitative approach to frequency (e.g. 80% of the time) we have found that by using a number it creates a pseudo quantitative performance measure requiring a recording and reporting system to collect quantitative performance data. This is not the intent of a qualitative performance measure. Accordingly, we have established the following qualitative description of attribute frequency:

• Always – every time, without exception;
• Often – generally, most times;
• Sometimes – occasionally, now and again; and
• Not Always – whenever an exception has been observed.

It is essential that in defining the qualitative performance levels we don’t inadvertently make them quantitative. For example, if the qualitative performance measure is Responsiveness, it is critical that Good is not specified as “80% of enquiries answered within 48 hours”. This is a quantitative performance measure. Instead, by using the qualitative frequency descriptors above this can be changed to “Good = the contractor OFTEN answers enquiries within 48 hours”.

While the difference may seem small, the intent of a qualitative performance measure is to make sure both buyer and seller have a performance discussion as opposed to accurately and definitively measuring performance. Accordingly, qualitative performance measures should not be related to financial rewards and remedies, but rather non-financial rewards and remedies. Given this, subjective performance measures are typically non-payment performance measures; that is either Strategic Performance Measure (SPMs) or System Health Indicators (SHIs).

Table 1 below provides an example of this approach to defining both attributes and frequency to define the scoring process and performance levels of a subjective performance measure.

Table 1 : Example of Scoring Attributes of a Qualitative Performance Measure

Aggregating Attributes

Finally, in setting our qualitative performance levels, where the qualitative performance measure has more than one attribute (as shown in Table 1 above), we need to decide how to aggregate the attributes to produce an overall performance score.

The main method used is deciding how many attributes at what performance score would lead to an overall superior, good, fair or poor, and how this changes as the number of attributes change.

For example, assuming superior in included in the scoring method, we would usually set this as “at least X of the Y performance attributes ranked as “Superior” and no single performance attribute ranked lower than Good.” In this case, by including both frequency requirements it ensures a high average performance against the attributes and no single ‘poor’ performers. Additionally, we may also set poor performance as two or more attributes assessed as poor. Table 2 below provides an example of this approach to scoring noting there were four performance attributes for this example.

Table 2 : Example of Aggregating Attributes of a
Qualitative Performance Measure

Summary

Qualitative performance measures can add significant value to the performance exchange between buyer and seller if the performance measures are designed properly using clear descriptions for the attributes and frequency. Given this, subjective performance measures are typically non-payment performance measures; that is either SPMs or SHIs.