Perspectives · Perspectives

A Warning about Models

Peter Milsom November 14, 2018
Due to the complexity of sustainability and social responsibility, we use a lot of models. One of the key lessons learned though is that "colouring by numbers does not guarantee success". It is important to realize the limitations and constraints of models, and the fact that they should always be re-evaluated, particularly when new information comes to light. This is important for all of us no matter what side of the debates we are on.

How Reliable are Models?

As a "methodologist", one of my favourite phrases is “all models are wrong… some models are useful,” by George Box, an industrial statistician.  

Why do we use Models?

Some people like prose, some like models, some like other things to describe and understand things. I like models personally. As Jay Forrester points out below, we organize our understanding of the universe in models to help us understand and interact with the real system or real world.    

How should we Continuously Evaluate Models?

However, using the scientific method, we must continuously re-evaluate and test our models.     From the 'Institute & Faculty of Actuaries' I appreciate the following lesson learned for dealing with models (and methods) in the real world:
Al models are deliberate simplifications of the real world. Attempts to demonstrate a model’s correctness can be expected to fail, or apparently to succeed because of test limitations, such as insufficient data... When a model fails, a variety of narratives may be offered to describe what went wrong. There may be disagreements between experts about the causes of any crisis, depending on who knew, or could have known, about model limitations. Possible elements include:
  • A new risk emerged from nowhere and there is nothing anyone could have done to anticipate it - sometimes called a “black swan”.
  • The models had unknown weaknesses, which could have been revealed by more thorough testing.
  • Model users were well acquainted with model weaknesses, but these were not communicated to senior management accountable for the business
  • Everyone knew about the model weaknesses but they continued to take excessive risks regardless. (Maggid, 2017)

CONCLUSION

The lesson learned out of this is to not believe completely in models, but to continuously test them out. They aid in communication, awareness and understanding, but they may not provide a complete picture of the real world and all of the possible permutations.  

REFERENCES

Maggid, Joshua. (April 16, 2017). All Models are Wrong. Actuary Info. http://actuary-info.blogspot.com/2017/04/all-models-are-wrong.html