Professional model audit: going beyond methodology

Embedding the right methodology within a rigorous quality management framework is key to delivering a professional model audit service.

In our previous article, we explored how ethics drive our decision-making. Now, we turn to the importance of embedding the right methodology within a rigorous quality management framework.

Pioneering the shadow modelling approach

Traditionally, financial model audit methodologies have relied on an inspection approach known as “cell-by-cell” review. This process involves model auditors meticulously checking a representative sample of cells in each row of a spreadsheet for errors or inconsistencies. While thorough, the main issue with this approach is that it does not deliver complete assurance. You can ask an analyst to do a cell-by-cell review, but you have no guarantee that they have spotted all the errors. To obtain more assurance, you can ask another analyst to re-perform the review, but even then, you have no proof that all errors have been captured. Studies (Panko, R., 2000) suggest that such techniques may identify only around half of all errors.

Recognising the limitations of such traditional methods, we pioneered a shadow modelling approach. Instead of relying on inspecting each cell individually, we built an independent template model—our “shadow”—that enables us to mirror the client’s model in purpose, scope, and granularity. In simplified form, we take the inputs from the client’s model, input them into our template, and compare the results. We then investigate each discrepancy until we fully understand it. This may then require an explanation of, or an update to, the client’s model. This process continues until the client’s model and the shadow model fully reconcile.

There are two main advantages to this method:

1. It is less monotonous: a major issue with undertaking a cell-by-cell review is maintaining concentration levels. There is only so much coffee you can drink! Shadow modelling is more like solving a puzzle, which helps engage and maintain the interest of the analyst.
2. It provides end-to-end assurance: if the models do not match, we know we have a problem.

All that said, there is nuance to this debate. Most established model auditors recognise the importance of some degree of shadow modelling, and we recognise that some elements of inspection are necessary to properly manage the risk of model error. Indeed, according to our own methodology, inspection is used as a complementary procedure when appropriate.

The building blocks of robust quality management

We have been using the shadow modelling methodology for over 17 years. It has proved to be highly reliable and is a key reason for the success of our team. However, in growing our dedicated team from just two to now around 100, we understood that the right methodology would not be enough in the context of our operations being scaled up. We therefore embedded our methodology within a rigorous quality management framework, which has provided the basis for being able to provide a consistently high level of quality to our clients.

Our quality management process is designed around the principle of understanding all the risks inherent in our model audit process. To do this, we start by asking ourselves, “What could possibly go wrong?” – because it probably will! We then design relevant controls to ensure each risk is managed. Finally, these controls are independently tested on a regular basis to ensure they are sufficient and effective.

The following are examples of how this approach is embedded in our process:

Comprehensive risk management: We systematically identify potential risks, such as an analyst misunderstanding a client’s model or failing to address certain elements of our review scope. Each of these risks is managed through targeted controls. For example, only analysts who have been accredited through rigorous training are allowed to tackle more complicated tasks.

Multi-level review: Our work undergoes multiple layers of review. At a minimum, the project manager reviews the reconciliation findings, followed by a further sign-off from a partner. The partner may in turn consult more widely if they have concerns. This structured oversight ensures that no single point of failure can undermine accuracy.

Parallel reviews for high-risk documentation: For more complex or higher-risk tasks, we employ parallel reviews, where two team members independently review the work. The results are then compared by the project manager, ensuring any discrepancies are identified before the final partner review.

Having the right methodology is undeniably important. Our shadow modelling approach exemplifies how innovation can enhance efficiency and accuracy in financial model auditing. However, without rigorous quality management, even the best methods can fail to deliver the desired results.

To briefly summarise this series on the professional approach to model audit so far: ethics form the foundation, and employing the right methodology within a robust quality management process builds the structure that supports our mission to deliver exceptional service.

As we continue our series, we will explore an increasingly critical aspect of our practice—data security—and how protecting client information is essential to delivering a professional model audit.