Choosing a Good KPI? (Part 1)

Not everything that can be counted counts, and not everything that counts can be counted.

Albert Einstein

One question we often get is what is a good performance measure? While you could simply Google “performance measure” (or more typically “KPI”) and this will give you a surprising number of responses.  But a word of warning before you click those links.

While performance measures are central to a Performance Based Contract (PBC), and the associated Performance Management Framework (PMF), instead, we would ask you to think about the intended purpose of the performance measure. What do you want it to do?

So before we choose our performance measures we need to look at the role of performance measures in a PBC? Kenneth G. Charron in his 2006 article on “Why KPIs Belong in Supply Chain Contracts” suggests that:

  1. Performance Measures establish set up the buyer’s expectations in terms of performance
  2. Performance Measures can provide an incentive for the seller
  3. Performance Measures can offer an objective way to assess performance
  4. Performance Measures result in more consistent performance
  5. Performance reports will enable the buyer to compare the performance (Benchmark) of one seller to other sellers of the same services—past, present, and future
  6. Performance reports can help identify the cause of on-going service failures
  7. Performance reports can increase buyer communication and improve service over time
  8. Performance Measures can help organisations avoid or reduce the number of disputes and the resulting costs

Alternatively, having run many PBC courses and workshops we ask this same question with answers generally focused on:

  • Reducing contract price
  • Improving contract performance
  • Adjusting contract payment based on seller performance
  • Understanding commercial issues

But is this true?  Do simply having performance measures lead to these outcomes (e.g. improved performance)?  You have probably seen contracts with performance measures in them.  Did the existence of performance measures lead to better outcomes and was it simply having performance measures, or were there other aspects as well?

The intent of performance management is to ensure that requirements are consistently being met in an effective and efficient manner regardless of whether this is related to the performance of an organisation, process, employee, etc.  Performance management includes the process of setting performance expectations, monitoring performance, measuring results, and appraising and rewarding satisfactory performance.  A best practice performance measurement system will, at a minimum, be composed of the following elements:

  • formal, organised structure for performance measurement and reporting;
  • clearly defined roles, responsibilities and accountabilities for performance measurement and reporting;
  • well documented data quality standards and expectations for performance information, including monitoring and quality assurance procedures, which are clearly communicated across an organisation; and
  • assurance arrangements which may, for example, be in the form of approved data dictionaries that include adequate documentation of data sources, collection methods, standards and procedures with clearly spelt our calculation/costing methods, assumptions, etc.

So while a critical element are the performance measures, they are not the only element needed to drive behaviours.  Indeed, our approach to PBC describes 2 distinct parts to a PBC; the requirements and the consequences. As you can see from the above list most people believe that a performance measure delivers both. But do performance measures do both?  For example, if a seller exceeds the buyer’s performance requirement and is eligible for an incentive payment, is it the performance measure that pays the incentive?

In summary, before you simply take that list of ‘KPIs’ from google it is critical you understand what you are using the performance measure for and how it links to the other parts of the PBC.

In the next part of this series, we’ll look at the process for choosing and checking you have the right performance measures.

Posted in the How | Tagged , , , , , | Leave a comment

Case Study – Setting Performance Levels

Early in my career at the start of Performance Based Contracting (PBC) as a Reliability Engineer I was asked to check a services contract that was delivering availability for a particular piece of equipment.

The buyer was concerned that the equipment was not meeting the required performance levels, in this case an availability level, specified in the contract and that the seller had agreed to deliver through competition.

On the other hand the seller, who had agreed to deliver to the performance level in the contract, was delivering the best possible performance.  Having completed an independent review, it was clear that the seller was delivering the best performance in the world (compared to other users) and was better than the design specification. However, it was still lower that the contracted performance level. So what do you do?

Discarding the do nothing option, the first option is for the buyer to enforce the contract by forcing the seller to delivery the performance. Unfortunately, in this case it was physically impossible for the seller to do better than they had done (world best practice) and the only outcome here would be contract termination. Interestingly, if the buyer looked for another seller through an open market Request For Tender (RFT) I questioned whether they could get better performance and price than they were getting.

The second option is for the buyer and seller to agree to ‘reset’ the performance levels in the contract through a formal Contract Change Proposal (CCP) to those being delivered. This allows the buyer to set the current performance level making sure of continued  delivery while assuring the seller of continued tenure by removing the threat of termination for the delivery of world-class performance; a win-win for both buyer and seller. Fortunately, the buyer and seller settled on option 2.

The lesson for this example is that in setting performance levels (see Setting the Performance Levels (Part 1, Part 2 and Part 3) for further information on how to sett performance levels in a PBC):

  1. The buyer – must make sure that the required performance level is possible for a seller to deliver to – there is no point contracting for something that can’t be done. This will simply lead to failed outcomes and a broken relationship.
  2. The seller – must make sure that they do not sign up to a required performance level that cannot be delivered. This may need the seller to either not bid for the contract or offer an alternative performance level with a rationale for why. Some readers may think that perhaps the contract was too good to lose. However, in my experience, some contracts are too bad to win and potentially threaten both the financial outcome and reputation of the seller.

Simply put, while I am not blind to either buyer or seller trying to maximise their commercial outcome, there is no point starting a commercial relationship that cannot succeed.

Posted in Case Study | Tagged , , , , | 2 Comments

Starting a Performance Based Contract (PBC) – Part 2

In the previous article (see Starting a PBC – Part 1) I looked at the need and benefits of using a transition period when starting a Performance Based Contract (PBC). In this article I will look at key considerations to using a transition period when starting a PBC including examining the effect of the transition period on the PBCs Risk-Reward balance.

During a transition period the performance measures can either be implemented (i.e. turned on):

  • all at once; or
  • phased in over differing times either individually or as groups (e.g. start all supply support performance measures once all the spares part have been bought and delivered).

Additionally, more complex PBCs can also include:

  • specific ‘transitional’ performance measures that only apply during the transition period, or
  • setting performance levels at a lower performance level during the transition period recognising the need to balance the buyer’s need to hold the seller accountable for some level of performance (even if lower than the final performance level) and the seller’s need to minimise commercial risk during the implementation.

As part of designing the transition period it is also important to balance risk and reward. If the seller is ‘protected’ against non-performance (i.e. paid 100% of the At-Risk Amount regardless of performance) the commercial risk is different and the buyer may want to vary the commercial rewards during the same period. One option many PBCs now use is to ‘shape’ the profit margin during the transition period to reflect the change in commercial risk. This approach is illustrated in Figure 1. Please note that the values used in Figure 1 are for illustrative purposely only and are not recommendations or represent the values in real PBCs.

Figure 1 – PBC Transition Period and Shaped Profit

Important factors to consider when using a shaped profit margin approach:

  • While profit shaping commences with the start of the contract the seller can choose an early exit from the transition period. For example, instead of having a 6 month transition period the seller could choose to only have a 3 month, or indeed no, transition period and access the full profit margin from day 1.
  • The variation in profit (fee) must be enough to motivate seller’s acceptance of the additional risk (if any). That is, if the difference is too small or the original base profit margin is too high, there may be insufficient reward for the seller to take on the additional commercial risk and exit the transition period.
  • Ensure that variation in shaped profit is linked to other Risk-Reward structures within the contract (e.g. contract extension, stop payment, termination, etc.).
  • Consider whether the profit (fee) varies with individual performance measures, groups of performance measures or all performance measures.

In summary, the use of a transition period when starting a PBC is a critical step in establishing the right recording, scoring, reporting and reviewing cycles. However, applying a transition period is not a simple as having 12 months without consequences. Instead, it requires both buyer and seller to understand what outcomes they are seeking in the use of a transition period and to tailor it to specifically meet this need.

Posted in Contract Management, the How, Transition Period | Tagged , , , , | Leave a comment

Starting a Performance Based Contract (PBC) – Part 1

One routine question when starting a Performance Based Contract (PBC) is whether to implement all aspects of a PBC on day 1 of the contract. Some people, especially buyers, advocate starting all aspects of the PBC immediately, especially the consequences such as changes to performance payment. Others feel that starting the PBC with a small ‘phase in’ period is preferable. So who is right?

With a couple of exceptions I believe there are benefits from starting a PBC with a small transition period. Benefits include:

  • getting buyer and seller staff used to recording, scoring, reporting and reviewing performance (e.g. my observation has been that using this process for the first time will uncover issues no matter how clear everyone (buyer and seller alike) believes the contract is); and
  • avoiding initial and unrepresentative performance due to:
    • human error; and / or
    • small sample sizes / small durations.

The only exception to the need for a transition period is where the seller has already been delivering the same product or service and there is only minor change in scope or performance levels. In this situation, there is no need for the seller to have a transition period. Indeed, this may be part of the buyer’s justification for remaining with the seller as opposed to re-competing the contract.

Importantly, a transition period should not be viewed as a research phase for:

  • selection of new, either extra or different, or deleting current performance measures regardless of whether Strategic Performance Measures (SPMs), Key Performance Indicators (KPIs) or System Health Indicators (SHIs);
  • changing the weighting of the current performance measures; or
  • defining or changing the contracted performance levels for any of the performance measures regardless of whether SPMs, KPIs or SHIs.

However, where the product or service is so new there is no established performance baseline, the buyer may include a specific transition period to determine this performance level under actual contracted conditions (i.e. in the hands of the user in the field). This is very rare and the transition period would not extend past 12 months in duration. That said, to protect both parties and to prioritise the relationship at the start of the contract, I would always recommend including a minimum of 1 reporting period transition period regardless.

Typically the transition period will last for the first 1 or 2 reporting periods. For example, if performance is reported by the seller and reviewed by the buyer on a 3 monthly basis (e.g. quarterly), it would be common to include a transition period for the first 2 reporting period (i.e. 6 months). During the transition period the buyer and seller record and report performance as they would normally in the contract, however, there are no or “limited” consequences for both positive (e.g. incentives) and negative (e.g. abatements) variation in performance. This is illustrated in Figure 1.  Please note that the values used in Figure 1 are for illustrative purposely only and are not recommendations or represent the values in real PBCs.

PBC Transition

Figure 1 – Effect of a PBC Transition Period on Payment

The reason I have included “limited” consequences in the above description is that most PBCs will keep limited commercial consequences such as the right to terminate for other circumstances during the transition period . Indeed, it would be rare that during a transition period that a PBC did away with all commercial consequences for either buyer or seller.

In the next article, I will look at key considerations when using a transition period when starting a PBC including examining the effect of the transition period on the PBCs Risk-Reward balance.

Posted in Contract Management, the How, Transition Period | Tagged , , , , | 2 Comments

Performance Based Contracting (PBC) Book

With the imminent start of 2018 many of us are thinking about our New Years resolutions, including me. One of my resolutions is to have finished a Performance Based Contracting (PBC) book by 1st July 2018. Having already worked for over 1 year on this project I hope you will help me focus and make sure that I keep my promise!

Posted in Book, the How, the What, the Why | Tagged , , , , , | Leave a comment

Happy 2017 Holidays to all the Performance Based Contracting (PBC) Blog Readers

At this joyous time of year, we are grateful for the time we can spend with our own friends and family, and we wish you all abundance, happiness, and peace in a New Year filled with hope. Happy holidays!

Posted in Admin, Holiday Message | Tagged , , , | 2 Comments

Reliability in Performance Based Contracts (PBCs) – Part 2

In the previous article (see Reliability in PBCs – Part 1) we defined how reliability of a system or service could be used in a Performance Based Contract (PBC) including defining whether we were measuring success or failure, and whether the failure resulted in a mission [critical] failure or not (defined as a logistics failure). In this article we are going to look at the detail needed to measure reliability from a contractual perspective.

Previously we defined those elements that would result in either a mission or logistics failure. However, from a contractual perspective and linked to performance management, who is attributed to (owns) this failure, Buyer or Seller? Typically, this refers to either a chargeable (to the seller) or non-chargeable (to the buyer) failure. Unfortunately, determining whether a failure is chargeable or non-chargeable is not simple, especially when dealing with complex systems and commercial arrangements (e.g. multi-party contracts). As illustration of this consider the following scenario.

The buyer receives serviceable (working) vehicles from the seller that are then driven by the buyer’s employees to deliver buyer’s goods. The seller is responsible for ensuring that there are enough vehicles available to the buyer with all maintenance completed and the vehicles are “roadworthy” (i.e. no safety or mechanical issues such as working lights).

Now consider the buyer has failed to make a delivery due to the vehicle not working. Is this a chargeable failure? If the failure was due to a mechanical issue (e.g. engine not starting) then this is simple; chargeable failure to the seller. However, what if the vehicle was in an accident that was the fault of the buyer’s employee? Or the buyer’s driver did not put fuel in the vehicle?  What then? What if a third-party caused the accident (e.g. neither the buyer nor seller)? Alternatively, what if the vehicle was operating outside it’s normal role (e.g. carrying twice the maximum design weight making the brakes less effective) or in a different environment (e.g. extreme cold resulting in an icy road). What now? Is the failure chargeable or non-chargeable?

In a previous article (see Setting the Performance Levels (Part 3)) I discussed criticality of Configuration, Role and Environment (CRE) in setting performance levels. In failure sentencing (the name given in the Reliability Engineering community for this process), we should also consider the CRE.  While it is possible to transfer all risks from buyer to seller, in many circumstances this risk transfer is unaffordable for the buyer.

The alternative is to specify all these exclusions in the PBC. Indeed many years ago my colleagues and I would try, sometime resulting in up to 3 pages of possible exclusions like the ones above. However, explicitly considering and documenting is neither effective or efficient since it is impossible to capture all possible event.  Additionally, the drafting and management cost to both buyer and seller becomes too high.

To avoid this in recent years my colleagues and I have started either:

  • defined general sentencing principles (as opposed to specific rules); or
  • used “all in” performance measures that are not adjudicated (e.g. no failure sentencing is undertaken). See When a KPI is not a KPI for further details.

The key to which of these 2 types you use is highly dependent on whether the commercial relationship can include “all in” performance measures as part of the performance measures hierarchy. In my experience many contracts are more “master-servant” in nature (i.e. the buyer will tell the seller what to do and how to do it) as opposed to a more partnering style that emphasises a “shared destiny” approach.  However, where the commercial relationship is more partnering and collaborative in nature, the “all in” performance measures, especially when linked to incentive (see PBC Incentive Regime Part 1 and Part 2), are very powerful drivers of positive outcomes for both buyer and seller.

In summary, like availability performance measures, reliability performance measures should be included in many PBC performance measure hierarchies as they represent the goal of many PBC buyers and sellers. However, care must be taken when designing these reliability performance measures to make sure that they accurately reflect the need and take into account the specific CRE of the commercial arrangement. Moreover, unless the commercial relationship reflects a more partnering approach, the use of “all in” performance measures will be difficult to include.

Posted in Performance Measure, the How | Tagged , , , , , , | 1 Comment