Internal Model Validation

31 Pages·2017·0.35 MB·English

by  Patel, Alpesh

Internal Model

Validation

Model Validation Guidance

July 2017
THIS PAGE IS INTENTIONALLY BLANK

2
Contents
1 Introduction 5
1.1 Purpose 5
1.2 Scope 5
2 Executive summary 6
3 Validation report format 7
4 Components of validation 8
4.1 Introduction 8
4.2 Independence 8
4.3 Risk coverage and risk ranking 9
4.3.1 Risk coverage 9
4.3.2 Risk ranking 9
4.4 Overview of the validation cycle 10
4.4.1 Analysis of validation test results 10
4.4.2 Escalation 11
4.5 Pass/fail criteria 11
4.6 Expert judgement 12
4.6.1 Expert judgements used in the internal model 12
4.6.2 Expert judgement used in validation 13
4.7 Limitations 14
4.8 Targeted validation 14
4.8.1 Reliance on previous validation 15
5 Key technical validation tests 16
5.1 Sensitivity tests 16
5.1.1 Deterministically varying a set of assumptions (“ST-1”) 16
5.1.2 Plausible alternative set of assumptions (“ST-2”) 16
5.2 Stability tests 17
5.3 Stress and scenario tests 18
5.4 Reverse stress tests (RSTs) 19
5.5 P&L attribution 20
5.6 Testing against experience 21
5.7 Benchmarking 21
5.8 Other tests 22
6 Thematic findings 23
6.1 One year validation 23
6.2 Dependencies and aggregation 24
6.3 Outwards reinsurance 25
6.4 Catastrophe risk 26
6.4.1 Model version 26
6.4.2 Model completeness 26
6.5 Operational risk 27
6.6 Credit risk 27
6.7 Events not in data (ENIDs) 28
6.8 Special Purpose Arrangements (SPAs) 28
Appendix A: Reverse stress tests: frequently asked questions 30
3
THIS PAGE IS INTENTIONALLY BLANK

4
1 Introduction
1.1 Purpose
This document provides guidance to managing agents in respect of internal model validation
requirements under Solvency II. It builds on previous Lloyd’s guidance and findings from
Lloyd’s review of the market’s validation reports.
This model validation guidance should be read in conjunction with the Lloyd’s Minimum
Standards, previous Lloyd’s guidance and workshops on model validation for a complete
understanding of the regulatory requirements in the Lloyd’s market. These can be found on
the Lloyd’s Model Validation section on Lloyds.com.
1.2 Scope
This document offers guidance on validation of the methods, assumptions and expert
judgement used in the internal model.
The following areas are not covered in detail in this document; additional guidance has been
provided by Lloyd’s in the sources listed below:
- Targeted validation
- Data
- Catastrophe risk
- Major model change
- Lloyd’s Minimum Standard 5: model scope, change and use
- Lloyd’s Minimum Standard 6: modelling, design and implementation
- Lloyd’s Minimum Standard 7: validation

5
2 Executive summary
The overall purpose of the validation exercise is to ensure that the internal model provides a
realistic and robust assessment of all the material risks faced by the syndicate. Validation is
an iterative process of identification of model limitations and the implementation of
improvements. The most appropriate validation tools for achieving this task will vary by
syndicate, depending on the size and complexity of its risks. This guidance is intended to
assist managing agents with the selection and application of the validation tools most suited
to their syndicates’ risk profiles.
Regulatory expectations have also been evolving; the Supervisory Statement SS17/16 from
the Prudential Regulatory Authority (PRA) discusses model validation in section 7 and the
internal model change policy in section 9.
Managing agents have made good progress in embedding the validation process within their
business since Solvency II was implemented. Some of the improvements noted by Lloyd’s in
the latest round of validation report reviews were:
adequate management level information was included in validation reports;
-
limitations of the model and validation process and failures in validation tests were
-
appropriately highlighted in the validation report;
the purpose and use of validation tests was clearer;
-
more clarity in the governance structure and escalation process was noted;
-
generally, the pass/fail criteria for quantitative validation tests were more explicit.
-
Based on the latest validation report reviews, Lloyd’s is advising agents to give special
consideration to the two areas listed below. These are both discussed in detail later in this
guidance.
1. Reverse stress testing: several inconsistencies were identified in execution of this
particular validation test. Agents should aim to define the scope and execute the test
appropriately in line with Lloyd’s guidance;
2. Structure of the report: in some cases the validation report did not include sufficient
technical detail on certain topics (e.g. expert judgement, sensitivity tests, one year
SCR, P&L attribution) to assessment of regulatory requirements. In addition to the
main validation report (provided to the Board), this may be submitted either as part of
a separate (more technical) report or as appendices to the main validation report.
The current state of model validation in the market is deemed to be well-progressed and
embedded within agents’ operations. This latest guidance is intended to continue the
advancement of model validation and improve efficiency and effectiveness of the process.
Lloyd’s intends to move to a more targeted approach to validation of material risks faced by
the syndicate. Further details can be found in section 4.8.

6
3 Validation report format
A good validation report should (at the very least):
- be concise;
- aim for a top-down view of validation;
- not exclude relevant quantitative details of capital and validation;
- be treated as a Board report;
- focus on material risks;
- emphasise failures;
- outline the limitations to the validation process.
This is a list of suggestions and should not be treated as an exhaustive list of requirements.
The technical details of validation tests and process are also required to be documented.
Lloyd’s has recommended different approaches to documenting and reporting details of a
validation process in previous workshops (validation workshop 2016 and validation workshop
2014).
In the latest round of reviews of agents’ validation reports, Lloyd’s observed a marked
improvement in the size and content of the main validation report. Most reports successfully
summarised the key validation findings in a concise manner. However, in some cases, the
level of quantitative detail was not adequate. It is important to include sufficient quantitative
detail, as the Board must have sight of key details to allow them take strategic and financial
decisions.
Lloyd’s is required to ensure, for regulatory compliance, that validation tests are carried out
consistent with the guidance issued. Lloyd’s identified that technical details are sometimes not
included in the regulatory submissions. This has led to challenges and required additional
work by those agents.
To address this, detailed information on the validation tests, audit trail and other technical
details may be included in a separate (more technical) report or as appendices to the main
report. This would ensure a concise validation report, summarising key details of the process,
is maintained. The main validation report should still retain relevant quantitative information
that helps identify possible limitations in the process or model. The additional report (or
appendices) would provide evidence that would assist in assessment of regulatory
compliance.
Throughout this guidance, the terminology ‘validation report’ is intended to specify the Board
report, while ‘validation pack’ is the regulatory submission to Lloyd’s (including both the
validation report and the additional supporting technical documentation).
As part of the validation pack, agents are requested to document how feedback from Lloyd’s
(and other stakeholders) during the previous validation cycle have been considered as part of
the development of the latest validation cycle.

7
4 Components of validation
4.1 Introduction
It is essential for the managing agent, for Lloyd’s, and for compliance with Solvency II that
syndicates’ internal models are thoroughly validated. The requirements for model validation
under Solvency II are extensive, and meeting them has brought many challenges for the
industry. This is especially true for Lloyd's due to the unique and complex nature of the risks
written in the market. Managing agents have invested significant time and effort over the last
few years in order to meet the requirements of Solvency II, including those for validation.
The validation exercise consists of many components which are covered in more detail in the
following sections. The key high level validation requirements include:
independence in order to ensure objective challenge;
-
a process to ensure that all material risks are covered in the model;
-
a risk ranking exercise and a demonstration that the validation effort has been
-
proportional to the materiality of the risk;
validation tools and tests to be well defined and appropriate for the risks being
-
validated;
outcomes of tests to be clearly explained and justified, and the path from a “fail”
-
outcome to escalation and model change (if appropriate) should be clearly mapped;
and
documentation of the validation process and the reasons for the steps taken.
-
4.2 Independence
Validation requires objective challenge. Agents should be able to demonstrate that the
individuals responsible for validation have sufficient independence from the design, build,
parameterisation and implementation of the model component being validated. They may
make use of tests carried out by those responsible for the modelling, but not rely entirely on
them.
There is no requirement for external validation to form part of the validation process. A
number of agents do however use third parties in the validation process in order to
supplement internal resources.
Where reports prepared by third parties (both internal and external) are used in the validation
process, the agent should make clear:
the scope of the work performed;
-
the reliance placed on the work; and
-
why the validator feels able to rely on the work performed by the third party (this may
-
take the form of the experience and qualifications of the external parties being set
out).

8
4.3 Risk coverage and risk ranking
4.3.1 Risk coverage
The objective of the validation process is to test that all material risks to the syndicate are
adequately assessed in the internal model. The initial steps in the validation process should
therefore be a gap analysis to test whether all material risks are indeed covered. A typical and
acceptable way to do this would involve:
an identification of risks to the business;
-
an identification of which of these risks are not covered by the internal model; and
-
an assessment of whether the risks not covered are material.
-
The risk identification process should identify all sources of loss to which the syndicate could
have non-trivial exposure. This process should not be restricted to insurance risks; it should,
for example, include considerations such as the terms and conditions of the cover issued,
data and operational systems, the current legal environment, recent market experience, and
so on. It should also take into account the possibility of new sources of loss not experienced
by the syndicate or market in the past, for example emerging risks. The P&L attribution
exercise has historically been useful in identifying risk coverage gaps – see section 5.5 for
further details.
If material risks have been excluded as a result of this gap analysis, the model needs to be
amended to include them.
Many agents have maintained a risk register as a way of identifying risks faced by the
syndicate. Lloyd’s considers this to be an appropriate approach under Solvency II. The format
of the register should be the one most suitable to agents. The risk register may also be used
to identify which risks are currently “in” or “out” of the internal model.
The risks not covered by the internal model should be assessed for materiality to the
syndicate. This requires explicit criteria for assessing materiality and/or risk indicators for
determining materiality. It is agents’ responsibility to determine and describe the materiality
threshold for their syndicates. The materiality threshold should be closely linked to those for
major model changes.
Historically, some of the largest losses to the market have come from sources of loss that
were only partially modelled or non-modelled. Examples include asbestos, multi-year reserve
deteriorations, and the World Trade Center terrorist attacks. For this reason, Lloyd’s considers
a comprehensive risk identification process to be the basis for sound validation.
Lloyd’s recommends that agents develop a comprehensive process for identifying potential
risks to the business and assess their materiality on an ongoing basis (at least annually). This
should cover changes in the risk profile (for example, an increase in cyber risk), and provide
re-affirmation that risks previously assessed as non-material remain so.
4.3.2 Risk ranking
The risk coverage exercise described above should be proportional to the size and complexity
of the business and confirm whether all material risks are captured. The purpose of risk
ranking, in the context of validation, is to identify those areas requiring the most extensive
validation. Risk ranking should therefore be one of the first steps in the validation of the
internal model. In addition to this, agents can also perform risk ranking at the end of a
validation exercise in order to check that the ranking is as expected at the conclusion of the
process.
9
Agents should be able to produce rankings consistent with the groupings used to manage the
business (for example, by class of business within reserve risk or by currency/asset class
within market risk). It is not necessary to rank every risk component of the internal model,
although the methodology should have the capability to do so. The ranking methodology may
be approximate, but should be broadly consistent across risk types.
There is overlap between risk ranking and the assessment of risk coverage described above.
Both can be done using approximate metrics, such as the materiality of the risk being
considered.
In this context, agents should view risk ranking as a tool for making the validation process
more efficient. It will enable them to allocate resources to areas which are quantitatively or
qualitatively material to the internal model.
Lloyd’s recommends that the validation policy includes a description of how the outputs of the
risk ranking exercise are used within the validation process.
4.4 Overview of the validation cycle
The validation cycle is at the centre of the validation process. It is comprised of four steps:
(1) the application of the validation test or tool;
(2) the analysis of test results;
(3) the escalation of test results to appropriate individuals in the business; and
(4) the implementation of any changes necessitated by the validation test outcome.
Definition and implementation of the validation cycle has been embedded in the validation
process through multiple iterations. Agents have provided relatively clear descriptions of steps
(1) and (2). Many agents provided sufficient information on step (3), but a practical challenge
often is to understand the right level of detail of the governance process that should be
included in the report.
Details of governance are often included in the validation policy. Adding a summary in the
validation report and clearly signposting where a fuller description might be found may
overcome challenges associated with step (3).
It can be a challenge to demonstrate the escalation process as many issues identified will
have already been addressed during the validation cycle and are therefore resolved by the
time the final validation report is completed. Examples of validation test failures and the
process followed for such cases is one way to evidence the governance structure in place.
Finally, Lloyd’s recommends that each validation test shows evidence of the four stages of
the validation cycle. In particular, a test outcome and the reasons for that outcome should be
clear.
4.4.1 Analysis of validation test results
The results of validation tests should be justified with clear rationale in conjunction with the
pass/fail criteria discussed in section 4.5; this ensures the effectiveness of the test. A good
way to provide adequate rationale is to support qualitative conclusions with quantitative
analysis or vice versa; two examples are provided below to illustrate this.

10