2.3. Methods

Key messages

  • National-level MRE systems need to be agile and pragmatic, using methods that are consistent with the national context (including needs, priorities, resources and data availability)
  • A ‘mixed methods’ approach to MRE, which combines quantitative and qualitative information, provides the strongest basis for analysing and assessing adaptation progress and performance
  • Indicators play a key role in national MRE systems, and they are often created in an iterative and interactive process with experts and stakeholders (the UK and Germany are examples of this)
  • Qualitative methods play an important role in complementing quantitative indicators and in revealing critical contextual information which can help to explain the narrative behind the numbers
  • It is not necessarily the value of an individual indicator that needs to be considered, but whether the set of indicators provides a coherent and robust picture of adaptation progress.

comments (0)

Introduction

National-level MRE systems for climate adaptation are context specific; they are shaped by the drivers and purposes that led to their development (see Chapter 2.1) and reflect the national adaptation policy context and governance structures (see Chapter 2.2). Often MRE systems are designed to track adaptation policies and programmes that are focussed on specific climate risks or priority sectors pertinent to the country in question. As a consequence of these contextual factors, much of the emerging literature on adaptation MRE guards against ‘one-size-fits-all’ approaches (Spearman and McGray 2011). This places an emphasis on practitioners learning about, and reflecting upon, methods applied in a range of different situations in order to understand what might work best for them.

comments (2)

It should be noted that to date, many countries have tended to focus on the monitoring aspect of MRE which has led to a greater emphasis on the development of indicator sets to track progress. For example, the German MRE system was established with monitoring as the primary function, with a set of activities proposed to advance evaluation aspects at a later stage.  In Spain, the main focus of their work has been on monitoring the Spanish National Climate Change Adaptation Plan (PNACC) but less has been done on evaluating the impacts of the PNACC. However, more countries, including the UK and France, are now reaching the point of evaluating adaptation progress and performance so are now considering a broader range of methods.

comments (0)

The aim of this chapter is to read across emerging and established MRE systems to identify and examine the methods countries have employed and to identify lessons from the approaches used.  As adaptation MRE is a continuous process, there are valuable lessons for countries looking to refine and improve existing MRE systems as well as for countries at a developmental stage. The chapter is structured around the following main methodological themes:

comments (0)

  • The development and use of indicators
  • The development and use of methods to gather stakeholder perspectives
  • Analysing and assessing evidence: a mixed methods approach
  • Addressing the challenge of attribution

comments (0)

Adaptation indicators

Indicators play an essential role in the monitoring, reporting and evaluation of policies, programmes and projects. They are a critical means by which processes, outputs and outcomes are tracked and assessed. When applied to climate adaptation, they can act as signposts of progress and implementation, “providing clues and direction on how change is occurring and if outcomes are being achieved” (Climate-Eval Community of Practice, 2015). If selected carefully, indicators can provide a valuable way to monitor adaptation policy implementation, as well as changing vulnerability and resilience, on an on-going basis. This monitoring information can then provide an important source of evidence for evaluations at key points during the policy cycle.

comments (0)

National adaptation strategies and plans often provide the strategic direction for adaptation but do not always clearly specify outcomes and targets (GIZ, 2013; OECD, 2015). Consequently, MRE systems are often required to develop and assign indicators to these priorities. Indicators can allow for comparison at sub-national level, between sectors or across a range of climate-related risks. Such comparisons can help in identifying hot spots of vulnerability that, in turn, can lead to improved prioritisation of adaptation investments. 

comments (0)

In 2014, ten countries out of twenty-two reported that they were implementing, or developing indicators on climate impacts, risks and adaptation (EEA 2014). Table 2.5 provides further details regarding indicator development for these countries, plus other countries for which information could be gathered in 2015.  It highlights the prominent role that indicators play as countries begin to develop MRE systems.

comments (2)

Table 2.5: Indicator development within national level MRE systems in Europe

Country

Status of indicator sets and types of indicators being developed

Austria

An indicator system for monitoring and reporting on adaptation for 14 sectors outlined in the Austrian Adaptation Strategy has been developed. It includes 45 qualitative and quantitative indicators to monitor the process, the output or the outcome of adaptation interventions.

Belgium

In the planned NAP each action is escorted by at least one indicator of results. Some are qualitative, some are quantitative. These indicators will be used for the intermediary evaluation (half-life of the plan) and the final evaluation.

In the Flemish Climate Plan (already in place), a system is being set up for the annual monitoring of adaptation progress in the different sectoral policies.

Finland

Evaluations of the National Adaptation Strategy (2009, 2013) applied a five-step scale to indicate the level of adaptation in different sectors (see box 2.8). Efforts to develop suitable adaptation indicators were initiated in 2015 to support the monitoring and evaluation of the National Adaptation Plan 2022.

France

Annual monitoring of progress is undertaken for 19 areas and one cross-sectoral theme outlined in the National Adaptation Plan (2011-15). For each area and theme, an action sheet outlines one to six actions, each comprising several components that must be undertaken in that area, totalling 84 actions and 230 measures. These actions can be broadly categorised as i) production and dissemination of information, ii) adjustment of standards and regulations, iii) institutional adaptation, and iv) direct investment. (from OECD 2015)

Germany

Indicator system for reporting on climate change impacts and adaptation areas outlined in the German Adaptation Strategy (DAS). 102 indicators: 97 for impacts and adaptation; 5 overarching indicators (e.g. awareness of the public, research funding, international funding, funding for municipalities)

Hungary

No information available at the time of drafting.

Ireland

In the process of developing adaptation indicators.

Lithuania

The planned MRE systems will be indicator-based and linked to NAP (3 year cycle). It will focus on 6 main sectors. Indicators likely to be qualitative

 

Malta

No information specifically available on indicators. Monitoring of the National Climate Change Adaptation Strategy (May, 2012) is conducted through the screening of the Malta’s National Environment Policy (NEP) under the sections related to climate change.

Netherlands

41 indicators (qualitative + quantitative), judged qualitatively

 

Norway

MRE at national-level is at an early stage with the use of indicators still being considered. The Norwegian Environment Agency is in dialogue with 11 other sectors agencies – about CCA in their sector, if relevant indicators exist or if they have input to potential indicators in their sector.

Slovakia

No information available at the time of drafting.

Spain

In the process of developing an impact, vulnerability and adaptation indicator system.

Switzerland

Switzerland are not planning to develop any new indicators but to use existing data sets that provide information on adaptation and/or the development of climate related risks or vulnerabilities.  The status of indicators varies between sectors in the adaptation strategy (some are completed, some in development, some have not started to determine indicators). A final decision on the indicators to be used at national-level when evaluating the effectiveness of climate change adaptation has not yet been made.

UK

As part of their statutory role to evaluate the progress of the National Adaptation Programme, the UK Committee on Climate Change (CCC) has developed a set of 33 indicators aligned to the themes and adaptation priorities identified in the NAP. These indicators are complemented by research and analysis undertaken by the ASC and presented in the evaluation report and previous non-statutory progress reports.

comments (12)

Types of indicators

An indicator provides evidence that a certain condition exists or certain results have or have not been achieved and can be either quantitative or qualitative (EEA 2014).

European countries have developed a range of different types of indicators to inform adaptation MRE at national level. These indicators are usually grouped into sets according to climate risk (e.g. in the UK, separate indicator sets were formulated for flood risk and for water scarcity).

comments (2)

Indicator groupings are often aligned to the sectors or themes emphasised within key national adaptation policies such as the NAP/NAS (e.g. in Germany indicators are organised across 13 Action Fields and 2 Cross-sectoral fields to be consistent with the German NAS). As well as providing a useful link between MRE and policy, these indicator sets are also important in providing a balanced picture of progress and performance. As there is no single measure of adaptation, it is important that MRE systems utilise a combination of indicators in order to provide a nuanced understanding of progress in each theme or sector.

comments (0)

Based on experiences so far, indicator sets developed for national-level adaptation are predominantly quantitative in nature. They can be grouped into two non-exclusive categories, which present different ways of considering what they are trying to measure and why. These categories are explored in Box 2.6: 

comments (0)

Box 2.6: Categories of indicators

Indicators from an adaptation perspective

Indicators can be used to measure different aspects of climate change. These include:

  • climate impacts (e.g. changing flood frequency)
  • adaptation policy responses and actions (e.g. a change in water management policy)
  • vulnerability (e.g. rate of residential development on the floodplain),
  • realised climate losses (e.g. annual damage costs from flooding) 

Indicators from a policy cycle perspective

Indicators can be designed to measure a process being undertaken, a specific output to be achieved or delivered, or a broader outcome:

  • a process-based approach defines the key stages in a process that could realistically be expected to contribute to positive adaptation outcomes, without specifying those outcomes at the outset (e.g. indicators that illustrate a process is underway such as the formulation of an coastal adaptation planning committee)
  • an output-based approach follows the direct results of an adaptation policy or action, without assessing yet if these results actually lead to better adaptation outcomes (e.g. indicators that an output has been achieved such as ‘x’ km of upgraded sea defences)
  • an outcome-based approach seeks to define an explicit outcome, or result, of the adaptation action, indicating a reduction in vulnerability or better adaptive capacity (e.g. indicators that show a coastal community is now less vulnerable to coastal inundation).

comments (3)

In terms of the aspects of climate change and adaptation being measured, experts reported a wide range of indicators being applied. Indicators include those focused on future and realised climate change impacts, exposure, vulnerability, adaptation actions and responses as well as longer-term outcomes. Similarly, a mix of process, output and outcome indicators has been applied. An important message that can be distilled from this variety is that it is not necessarily the value of an individual indicator that needs to be considered, but instead whether the set of indicators developed for a specific sector or theme provides a coherent and robust picture of adaptation progress as a whole. The importance of combining multiple indicators reflects the lack of a single, universal indicator for adaptation, the multi-faceted nature of adaptation (meaning more than one aspect should be measured) as well as practical considerations such as data coherence and availability. 

comments (0)

Process indicators capture present-day contributions towards a long-term aim (Bours et al., 2014), making them useful for adaptation MRE, as outcomes of adaptation policies and actions often cannot yet be determined. Consequently, they are an important feature of many emerging MRE systems. The degree to which countries have focussed on process, output or outcome indicators varies. For example the Netherlands and Lithuania have, or plan to have, a strong focus on process indicators. In Germany, indicators have been developed to inform an improved understanding of the causes and effects (impact indicators) of climate change, its consequences and subsequent policy responses. Process indicators are included, but there is a greater focus on outputs and outcomes.

comments (2)

The challenge of developing useful outcome indicators is reflected in the limited number found in current MRE systems. Instead, a mix of process and output indicators is commonly used to understand progress towards broader outcomes. Outcomes are often articulated in terms of increased resilience or decreased vulnerability to a given risk or in a specific sector. The challenge then exists to test the causal link between adaptation policy responses and these broader outcomes. The UK approach aims to achieve this by combining indicator types in order to inform a broader understanding of progress towards outcomes; in which the preparedness of society to specific climate risks is used as an overall frame. This presents an ambitious approach that seeks to link the existence of plans and policies (process indicators), NAP actions (often a mix of process and output indicators) to changing vulnerability to critical climate risks (vulnerability indicators). Both quantitative indicators and qualitative information are used in order to understand progress towards outcomes. Examples of a mixed methods approach is explored in more detail in Boxes 2.8 and 2.9.

comments (0)

Benefits and challenges of developing indicators

Consultations undertaken with MRE experts highlighted the benefits and challenges of developing indicators. All experts consulted viewed indicators as an important tool for reporting on progress relating to adaptation policy. Quantitative indicators were seen as being attractive and practical as they can communicate clear and simple messages that are easy for policymakers to grasp. Some experts specifically referred to perceptions of objectivity and robustness that are often associated with quantitative indicators which make them particularly useful when communicating key messages regarding adaptation progress. Experts also felt that standardized, well-described indicators allow repeated measurement and the opportunity to create a time series, enabling the identification of trends. The use of pre-existing indicators (in some cases adapting them to purpose) is an accepted and pragmatic approach that brings advantages in terms of efficiency as well as providing multiple perspectives on adaptation.

comments (0)

Indicators are central pillars of most climate adaptation MRE systems at national level, however the experiences of countries highlight a number of practical challenges for practitioners. The availability of data is the primary challenge identified and, consequently, indicators often are supplemented by information gained through surveys and expert elicitation, as is explored later in this chapter. Data availability issues can include challenges related to lack of data, poor spatial coverage, lack of spatial and temporal coherence and the timeliness of new data. These issues are often reinforced by a tendency to use existing data sets that may have been developed for different purposes and are aligned to different policy cycles. Experts also recognised that applying pre-existing data sets as proxy indicators can raise questions regarding validity and attribution; is there a legitimate causal link between the indicator, climate change and policy implementation? This issue of attribution is explored in greater detail later in this chapter.

comments (0)

While indicators are clearly a valuable asset in the armoury of the policy evaluator, it is vital to remember they are only an indication of progress and performance. An over-reliance on indicators as a single, primary source of information on adaptation effectiveness and efficiency can lead to inappropriate assumptions regarding causality. The application of indicators can help to reduce the complexity to a manageable state by identifying and measuring essential components and relationships within the system (ASC, 2011) but this results in trade-offs in credibility, robustness and legitimacy (Miller et al., 2012). However,  while indicators allow a simple message to be conveyed, the aggregation of indicators can mean that some information is lost. This places greater importance on communicating key messages in ways that are clear but not overly simplified (this is explored further in section 2.4). 

comments (0)

Recognising the limitations of quantitative indicators is critical if they are to be used successfully. Used in isolation, indicators are not effective in revealing the narrative behind complex issues such as climate adaptation and rarely tackle the question of ‘why’ or ‘how’ adaptation is influencing vulnerability or resilience. They must therefore be used with care and, as explained later in this chapter, in tandem with other methods. 

comments (2)

The process of developing indicators

The design of indicator sets is usually the responsibility of a single organisation that often has broader responsibilities for adaptation and/or MRE, which could be a government ministry, a commission or committee or an environment agency (see chapter 2.2). At the same time, this process usually relies on the contribution of a range of stakeholders, including scientific experts, stakeholders within key sectors and organisations with existing responsibilities for monitoring and data collection. Based upon the experiences of European countries developing MRE systems, there are three main motivations for this wider engagement; firstly to ensure the indicators selected are ‘fit for purpose’ and are scientifically robust; secondly, to check the feasibility of data collection; and thirdly, to provide transparency and independence.

comments (0)

Most countries have taken a pragmatic approach by developing indicator sets comprising a collection of pre-existing indicators that together provide insights into impacts, vulnerability and the appropriateness of adaptation policy responses. This places further emphasis on establishing effective means of cooperation with a wide range of stakeholders in order to ensure that the indicators are appropriate and that data is available.  Box 2.7 highlights the systematic approach developed to develop an indicator system in Germany which emphasises the importance of transparency in the design and selection of indicators and the benefits of documenting the process to improve future indicator sets. It also illustrates the range of stakeholders and experts often involved in the process of indicator development.

comments (0)

Box 2.7: A systematic approach to indicator development in Germany

Key messages

  • A well-structured system for documenting the process of developing indicators can support the Monitoring and Reporting system as a whole. 
  • Indicator factsheets facilitate technical exchanges with the experts and support the technical and political agreement of the indicators.
  • Background papers ensure transparency and facilitate the development of additional indicators in the future.

A key element for the German Adaptation Strategy (DAS) is an initial progress report on the implementation and further development of the DAS, which is due at the end of 2015. To support this progress report an indicator-based Monitoring Report was published in May 2015. An Indicator System, developed through an inter-departmental process, therefore plays a vital role in providing information for both the monitoring and progess reports.

The DAS Indicator System constists of 102 indicators developed across 15 themes or action fields. Numerous experts from agencies at the Federal and State level, and scientific and private institutions participated in this process, and the final output received political agreement.  The process began by determining what needed to be monitored in terms of “Impacts” of climate change and associated “Responses” (adaptation measures or activities implemented). These were then collated and grouped into ‘Indication Fields’ aligned to specifc sectors or identifed as being cross-sectoral. The following criteria was applied to determine whether an indicator was suitable for the DAS Indicator System:

  • The indicator must be closely associated with climate change issues and adaptation.
  • The development of ‘Impact Indicators’ describe areas that are currently affected by climate change, or are expected to be affected in the future.
  • The ‘Response Indicators’ describe activities which support the adaptation process and, where applicable, they also describe developments which counteract this process;
  • Data must be available for formulating the indicators. The indicator must provide certainty that the data will remain available for the foreseeable future; and that it can be procured at reasonable cost and effort nationwide.

A documentation system, consisting of indicator and data fact sheets, supports the process of updating the indicator documentation, the data sources and the reporting process itself. Indicator fact sheets provide a justification of why an indicator was chosen, and identify weaknesses with regard to the interpretability, availability and comprehensibility of data and determine the areas of responsibility (remits) for updating. Data fact sheets (Excel files) contain both the data sets required for calculating or illustrating the indicators and the relevant metadata sets.

Ensuring transparency and enabling improvements to indicators in the future were important considerations in  the German approach to indicator development.  Comprehensive documentation of discussions and background papers helps to improve transparency and to facilitate the development of additional indicators in the future.  A “User Manual” has also been developed to provide guidelines for all work required for future updates of the indicators and for allocating roles among different contributors.

For further information: http://www.umweltbundesamt.de/publikationen/evaluierung-der-das-berichterstattung-schliessung

comments (2)

Stakeholder perspectives

Indicator-based systems are usually complemented with qualitative methods, including gathering views and opinions of stakeholders or particular communities of expertise. Nearly all countries acknowledge the benefits of such approaches, however there is no clear pattern in terms of the methods used. As with MRE of adaptation generally, in some cases consultation with civil society, sector experts and the general public required is a requirement of broader adaptation legislation (e.g. Malta consultation is stressed the enactment of the Climate Action Act, 2015) while in other cases it viewed as a voluntary, yet logical, means of gathering information. Stakeholder perspectives are perceived to play an important role in validating quantitative indicators and in revealing critical contextual information that can help to explain the narrative behind the numbers. 

comments (0)

Methods to gather stakeholder perspectives

Stakeholder perspectives provide predominantly qualitative evidence regarding the progress being made toward long-term outcomes. Emphasis is placed on stakeholders assessing adaptation progress and to identifying gaps, often from the perspective of a specific sector (e.g. infrastructure) or theme (e.g. biodiversity). These approaches are sometimes referred to as self-assessments, though often they will involve experts assessing progress beyond activities with which they are directly involved. Information derived from these assessments is usually collated and analysed by the body responsible for the MRE system.   

comments (0)

Methods commonly used for collecting stakeholder perspectives include surveys, workshops, in-depth interviews, consultations, or the establishment of expert panels or groups. There is no evidence of standardisation of these approaches and how they are applied amongst the European countries who have established MRE systems. However, these methods are often based around the principles of expert elicitation (the synthesis of opinions of technical and scientific experts) which is commonly used where quantitative data is insufficient or where data needs to be validated or contextualised.

comments (0)

The variety of ways in which different methods are used to gather stakeholder perspectives is evident when the use of surveys is examined. In Switzerland, a survey of adaptation progress among canton-level governments is being planned, which contains mostly closed question and a limited number of open fields. Such an approach may aid analysis, enabling greater comparability between responses, however this contrasts with a survey of sectors undertaken in France that was far less structured. The French approach has the advantage of gaining nuanced insights into the links between climate risks and adaptation policies and actions, and allowing a greater diversity of responses.

comments (2)

As the number of countries developing MRE systems increases, collaboration between countries is becoming an important source of expert knowledge. This means that those undertaking MRE of adaptation in other countries become important stakeholders for their counterparts and experiences elsewhere can inform the methods employed. For example, the UK Committee on Climate Change commissioned a study into the use of global and national indicators (Horrocks et al. 2012) and Switzerland has examined approaches used elsewhere. There are also examples of European countries engaging national experts from other countries in the review of their MRE approaches, which further supports the transfer of knowledge across national boundaries. As well as generating valuable information for this report, the Expert Workshop held in Copenhagen also played an important role in stimulating the type of knowledge exchange between experts.  

comments (2)

Benefits and challenges of gathering stakeholder perspectives

Generating information for MRE through the engagement of sectoral and thematic experts has a number of benefits. It can prevent an over-reliance on indicators, help to validate quantitative data and enable the exploration of the questions of ‘how’ and ‘why’. In turn, a deeper understanding of the causes and processes underpinning adaptation progress can reveal a clearer picture of causality and therefore support the validation of attributing outcomes to adaptation measures (or at least determining the contribution they have made). The significance of stakeholder perspectives in facilitating “improvements in understanding what is actually happening on the ground” was stressed by one participant in the expert workshop on MRE at national level held in Copenhagen[1].



[1]  ‘Monitoring, reporting and evaluation of climate change adaptation at national level’ held by the European Environment Agency on 24-25 March 2015

comments (0)

A number of other benefits stemming from these efforts to capture stakeholder views and knowledge to inform MRE have been observed by European countries. Involving key stakeholders and experts in specific sectors can help to foster commitment and improve learning on the side of stakeholders implementing adaptation actions, thus strengthening the implementation of adaptation policies and actions as well as informing MRE. Self-assessment approaches in Finland emphasised the fact that sectoral experts know their sector best, therefore they can provide practical insights regarding the enablers and barriers to adaptation that quantitative indicators cannot identify. In a number of examples the stakeholders targeted through self-assessment processes were those who implemented measures (e.g. France) or stakeholders that are specifically mentioned in NAP/NAS being evaluated (e.g. Austria). Engaging those best placed to enhance adaptation policy implementation in the MRE process ensures that MRE reflects local and sectoral contexts but also provides a valuable way of reflecting on practice and improving learning.

comments (2)

As with adaptation indicators, methods to gather stakeholder perspectives come with challenges. Surveys were seen as a useful tool, however, if questions are ‘open’ they can risk being misinterpreted which highlights the importance of providing clear guidance for respondents. There are also practical considerations regarding surveys such as the time and resources required to design, undertake and analyse a survey and the need to balance the benefits of a detailed survey with the time constraints of participants.

comments (0)

Clarity and transparency regarding the role of the experts or stakeholders being consulted is also important. Do they represent an individual expert voice, or do they speak on behalf of a sector or region? How representative are their views? Some sectors are very broad and members may have conflicting views on progress and appropriateness of adaptation efforts. In the same way that it is important not to depend on a single indicator, experts also stress the need not to rely on a ‘single voice’. There can be a lack of transparency regarding the process of formally incorporating expert and stakeholder views into MRE methodologies and how these views are balanced with information from other sources. 

comments (0)

Analysing and assessing evidence: Applying a ‘mixed methods’ approach

When analysing and interpreting monitoring and evaluation data it is evident that a combination of the quantitative and qualitative methods outlined above allows more effective triangulation of MRE information. Different data sources can be checked against each other to ensure that the overall narrative of adaptation progress is robust, consistent and contextualised.  A clear message from the Expert Workshop, reinforced by a recent OECD report (2015), is the need for blended or ‘mixed methods’ approaches. They can help to overcome some of the limitations of both quantitative indicators and qualitative data provided by stakeholders.

comments (0)

In order to gain an accurate picture of adaptation progress and performance at national-level, MRE systems often need to draw upon multiple data sources, across a range of scales and sectors. For example, a country may wish to understand adaptive capacity within government but also make use of existing data on, for example, changing ecosystems and habitats. Therefore a national level MRE system needs to be flexible, and be able to synthesise information from multiple sources. A system which utilises mixed methods is likely to be more agile than one reliant on a single source of data.

comments (0)

Experience from established MRE systems highlights that indicators play a prominent role, especially when supported by other methods. For example, in Belgium vulnerability assessments (using both qualitative and quantitative information) are combined with a set of ‘easy-to-use’ indicators which focus on the implementation of adaptation measures. In Norway, national-level MRE of adaptation relied on ‘bottom-up’ data from multiple sources rather than a pre-defined set of national-level indicators. Finland, which was one of the first countries to implement an MRE system, has combined reports on adaptation actions, self-assessments, workshops to gather expert views and a survey of regional adaptation (for more detail see box 2.8). Similarly, in the UK indicators and self-reporting by those responsible for managing key risks are used alongside expert judgement. The strength of any MRE system often lies in the ability to collate and analyse multiple sources of information into a consistent and robust evaluation.

comments (0)

Box 2.8: Evaluation of the National Adaptation Strategy in Finland using self-assessment and stakeholder perspectives

  • Self-assessments by actors responsible for the implementation of adaptation measures offer a cost-efficient method to generate insights on adaptation progress, and enhance learning among participating actors.
  • Collection of multiple stakeholder perspectives and use of different data sources facilitates balanced evaluation results.

The latest evaluation of the Finnish NAS in 2012-2013 built on a number of methods and information sources to assess the level of adaptation in different sectors, including self-assessment by government actors and collection of stakeholder perspectives through workshops, surveys and interviews.

As a first step in the evaluation process, a self-assessment was carried out by sector representatives in the National Coordination Group for Adaptation, who assessed the status of implementation of adaptation measures in their sectors. The self-assessment was supplemented with information collected in an expert survey. These experts were asked to evaluate the success of implementation of the adaptation measures, overall progress of adaptation in sectors, possible bottlenecks for adaptation and key needs for additional measures and tools to support adaptation. Additional data for the evaluation was collected in four thematic interviews, focusing on adaptation measures in the industry, health, insurance and built environment sectors, as well as a workshop directed at adaptation experts and practitioners.

Based on the aggregated data and stakeholder views, each sector’s level of adaptation was described as a position on a five-step scale of adaptation. This qualitative indicator combines multiple, primarily process-based elements relating to the implementation of the NAS, such as recognition of the need for adaptation in the sector, availability of knowledge on climate impacts, identification and status of implementation of identified adaptation measures, level of cross-sectoral cooperation, and level of mainstreaming of adaptation into regular processes and activities in the sector. Details of the different steps are described below.

Step 1

• Need for adaptation is recognised among a group of pioneers in the sector

• Little research is done yet on the impacts of or adaptation to climate change

• Some adaptation measures are identified but not yet implemented

Step 2

• Need for adaptation measures is recognised to some extent among decision-makers in the sector

• Impacts of climate change are known indicatively (qualitative information), taking account of the uncertainty involved in climate change scenarios

• Adaptation measures are identified, plans for implementation made and some measures launched

Step 3

• Need for adaptation measures is quite well recognised (majority of decision makers)

• Impacts of climate change are quite well known (quantitative information), taking the uncertainty involved in climate change scenarios into account

• Adaptation measures are identified and their implementation is launched

• Cross-sectoral cooperation on adaptation measures has started

Step 4

• Need for adaptation measures is widely recognised and accepted in the sector

• Adaptation is incorporated into regular decision-making processes

• Impacts of climate change are well known, within limits of the uncertainty linked to scenarios

• Implementation of adaptation measures is widely launched and their benefits are assessed at least to some extent

• Cross-sectoral cooperation on adaptation measures is an established practice

Step 5

• Adaptation measures under the Adaptation Strategy are recognised or otherwise implemented in the sector

In some sectors, the evaluation drew on the views of a handful of experts, but in most sectors multiple perspectives were collected from different stakeholder groups. Additionally, the evaluation utilised results of the NAS mid-term evaluation (2009) as well as results of international studies that have evaluated Finland’s NAS.

Finland’s National Climate Change Adaptation Plan 2022 (in English): http://www.mmm.fi/attachments/vesivarat/Fc3ezODON/mmm_ilmastonmuutos_eng_A4_v3.pdf (Annex II summarises key results and recommendations of the evaluation)

Evaluation of Finland’s National Adaptation Strategy (in Finnish only):

http://www.mmm.fi/attachments/mmm/julkaisut/tyoryhmamuistiot/2013/6MoQ7USVg/Ilmastonmuutoksen_kansallisen_sopeutumisstrategian_2005_arviointi.PDF

comments (0)

Box 2.9: Combining quantitative indicators with qualitative information to assess adaptation progress in the UK

Key messages

  • The UK approach provides an statutory, independent assessment of progress in delivering the National Adaptation Programme
  • The approach assesses how vulnerabilities to key risks are changing, as well as considering whether NAP actions have been delivered and whether plans and policies are in place. This provides a broader understanding of the changing context in which adaptation in occurring, which is critical for the assessment to be meaningful.
  • The UK assessment process makes use of indicators alongside a range of other data sources (including stakeholder perspectives) which are then analysed in order to provide an independent view of adaptation progress.  

The 2008 Climate Change Act requires the UK Government to conduct a Climate Change Risk Assessment and, in light of the risks identified, develop a National Adaptation Programme (NAP). The Adaptation Sub-Committee (ASC, as part of the independent Committee on Climate Change) is tasked under the Act to evaluate every two years the progress being made in delivering the NAP. This forms part of a larger report to Parliament on both climate mitigation and adaptation.

The adaptation assessment report follows the structure of the NAP focusing on six priority themes: the built environment, infrastructure, healthy and resilient communities, agriculture and forestry, the natural environment, and business. A cross-cutting chapter on local government is also included.  To evaluate the progress being made by the NAP, the ASC has considered three key questions for specific adaptation priorities identified under each theme:

  • Is there a plan? The ASC has assessed whether policies and plans in each area address the relevant climate risks. For example, the National Planning Policy Framework explicitly considers climate change and provides a basis for land-use planning decisions that account for current and future flood risks.
  • Are actions taking place? The ASC assessed the 371 actions listed in the NAP, as well as any other relevant activity that may be helping to reduce the impacts of climate change.
  • Is progress being made in managing vulnerability? To arrive at an overall assessment, the ASC has considered the available data to conclude whether vulnerabilities to climate change risks are increasing or decreasing.

The assessment is informed by a set of indicators that the ASC has identified and, in some cases, developed. These indicators measure any trends in changes to exposure and vulnerability over time, and observed impacts. For each indicator an assessment of the ‘direction of trend’ (increasing, decreasing or static) and ‘implication of trend’ (is the risk increasing?) is made.

The ASC uses information derived from the indicator set alongside a variety of other sources in order to assess adaptation progress.  These include a series of non-statutory annual progress reports that the ASC produced between 2010 and 2014 in the run-up to its statutory report in 2015. These reports examined specific risks and themes in detail and drew upon the expertise and knowledge of key organisations across relevant sectors.  The indicators used in the 2015 report were identified through these annual ‘deep-dive’ assessments.

The ASC widely consulted with key organisations as part of the statutory assessment process.  All organisations that had actions attributed to them in the NAP were contacted and given the opportunity to provide the ASC with an update on the implementation of their actions.  In addition, where significant data gaps existed, the ASC commissioned specific research projects. 

The ASC then analysed this combination of qualitative and quantitative data and came to a judgement with regard to each of the three key questions outlined above. A ‘traffic light’ system (red, amber and green indicators) was then used to highlight the progress for the key adaptation priorities in each thematic area of the NAP.  Figure 2.3 below from the statutory adaptation assessment report illustrates the overview of progress for adaptation priorities within the ‘Agriculture and forestry’ NAP theme.

Figure 2.3. Overview of adaptation progress for agriculture and forestry

Thirty-six specific, time-bound, recommendations were made within the assessment report where further progress is felt to be most important. The Government is required to provide Parliament with its response to these recommendations by 15th October 2015.

The “Reducing emissions and preparing for climate change: 2015 Progress Report to Parliament” can be found here: https://www.theccc.org.uk/publication/reducing-emissions-and-preparing-for-climate-change-2015-progress-report-to-parliament/

The ASC’s adaptation indicator set can be found here: https://www.theccc.org.uk/charts-data/adaptation-indicators/

As well as the report itself, the website contains links to the full list of the indicators used, the updates received on the 371 NAP actions, reports from research projects commissioned and technical annexes for each NAP theme that summarise the key trends identified by the indicators.

comments (0)

Addressing the challenge of attribution

Attributing outcomes to specific policy actions is a challenge for MRE in many fields, but is particularly difficult in the case of MRE of adaptation. Generating a plausible causal link from climate risks and impacts to climate adaptation policies and actions (processes and outputs), and to changes in vulnerability and resilience (outcomes) at national level is a major challenge. This stems partly from the long timescales and uncertainty associated with adaptation as described in the Setting the Scene chapter   of this report. Attribution is also challenging because of the sheer range of social, economic and environmental factors that can influence long-term outcomes, many of which are external to specific adaptation policies or actions.  No single solution exists for the ‘attribution challenge’, however a number of factors can be identified which may help MRE practitioners to address it.

comments (0)

Use of both qualitative and quantitative information

It is difficult to determine the existence of a causal relationship between a policy action and an outcome using only quantitative indicators. As previously mentioned, a mixed approach, making use of both quantitative and qualitative data sources, can produce a more comprehensive assessment of the relationships between policy actions and outcomes. It can also avoid misinterpretation of causes and effects, as evident in the example from the UK presented in Box 2.10.

comments (0)

Box 2.10:: A lesson in interpretation from the UK

In the UK the annual growth rate of [property] development in the floodplain is used as an indicator of vulnerability to flooding. Data shows that this growth rate decreased between 2008 and 2011, when compared to the rate for the period 2001-2008. Taken at face value, this could have been misinterpreted as a sign of successful adaptation policies. Yet when data on the annual growth rate of development in all locations is examined, it is apparent that this ‘slow down’ in flood plain development is part of a national trend; a reduction in rates of property development in response to the global economic crisis in 2008. In this example, it is therefore not possible, or correct, to link a changing indicator to adaptation policy effectiveness.

comments (1)

Clearer articulation of policy objectives

A number of experts at the MRE workshop stated that if aims and objectives had been more precisely specified within national adaptation polices and plans (NAP/NAS), it could have been easier to attribute outputs and outcomes to policies. Without this clarity MRE systems are required to ‘identify’ instead of ‘track’ a causal chain between policies and outcomes, which is a much harder task. Such precisely defined objectives can be useful, especially if they include an articulation of the assumptions made (possibly by developing a Theory of Change, see Box 2.11 below), thus enabling the MRE system to test the validity of these assumptions as well as tracking outputs and outcomes.

comments (0)

Ensuring adaptation initiatives are connected

Attribution may also be made easier if risk assessment, national adaptation policy and MRE processes are carefully coordinated and brought closer together. The tendency for MRE systems to be aligned to existing adaptation governance structures is useful in this regard. In the UK, the national Climate Change Risk Assessment, NAP and MRE of the NAP have all completed their first cycle; it is now easier to see how these elements could be better coordinated. 

comments (0)

Mapping expectations and assumptions

Attribution is easier to assess if expected relationships between climate change risks, adaptation policies and actions and outcomes (changes in vulnerability and resilience) are mapped out to begin with. This recognition has led to a growing interest in Theory of Change approaches for adaptation MRE (see Box 2.11), although few examples are evident at national level as of yet. 

comments (0)

Box 2.11: Theory of Change

Theory of Change is a critical thinking approach to program design, monitoring, and evaluation. This approach identifies a long-term outcome(s) then ‘works backwards’ to outline the building blocks, and the relationships between them, that would lead to the accomplishment of a long-term goal. Theory of Change explicitly identifies assumptions (‘if Y occurs, we expect X to happen’) enabling these assumptions to be tracked and evaluated. Theory of Change may be of particular use for national-level adaptation MRE as it can “tie together diverse projects and programmes into a coherent and strategic portfolio that enhances linkages across CCA sectors and scales” (Bours et al., 2014). The implementation of adaptation measures can be seen as a way of testing the hypotheses of the explicitly formulated ‘Theory of Change’ on adaptation.

See http://www.ukcip.org.uk/wordpress/wp-content/PDFs/MandE-Guidance-Note3.pdf for more information on Theory of Change

comments (0)

Some of the principles of the Theory of Change approach are applied in  the Netherlands where ‘signposts’ are used to help to assess whether the Dutch NAS is meeting the underlying conditions which have been identified as being critical to its success (see box 2.12 for further details). In Switzerland an Impact Model forms the basis of the MRE approach (see box 2.13) and this begins to set out the logic underpinning the flow from concept to outcome and impact, potentially allowing for a deeper analysis of assumptions and a better understanding of the adaptation process.

comments (0)

Box 2.12 Netherlands example

The Netherlands - The monitoring wheel to assess the state of climate adaptation in multiple sectors

Key messages

  • The monitoring wheel helps policy makers and stakeholders in assessing the progress around monitoring and evaluation of adaptation policies in sectors. This way achievements can be accounted and sectors can learn.
  • An innovative element of the monitoring wheels is the definition and evaluation of signposts. They determine the validity of choices made and as such provide a link between short-term decisions and the longer-term vision.
  • First applications of the monitoring wheel showed that flood protection is the most advanced policy area within the Netherlands for adaptation, whereas multiple other fields are still in early stages of development.
  • However, as also depicted by the monitoring wheels, much is going on in the other sectors, although not automatically assigned to climate adaptation.

In the Netherlands the monitoring wheel was introduced as a tool to assess the development and effectiveness of adaptation policies, measures and actions. It uses a colour-based system to depict for sectors that are relevant for the NAS the ‘state-of-play’ in steps of the policy cycle (inner cycle) and 16 underlying performance areas (outer circle). The basis for the filling the wheels for each sector is a list of 36 process, output and outcome indicators.

Innovative in this concept is the definition (step II.3) and evaluation (step III.7) of signposts. These signposts specify for each sector information that should be tracked in order to determine whether a plan is meeting the conditions for its success. These signposts can be climate (e.g. frequency of heatwaves, max. river discharge) and non-climate (financial resources, political conditions, socio-economic and technological developments) related. In addition, critical values of signpost variables (triggers) need to be specified beyond which additional actions are needed. Up to now this has been done for only few sectors (e.g. flood protection is based on a maximum discharge of the Rhine river of 16000 m3/s; chosen flood measures and actions are based on specific founding until 2050). Signposts determine the validity of choices made and help to determine adjustments that might be needed to the strategy and/or its policies. As such they provide a link between short-term decisions and the longer-term vision.

The monitoring wheels have been developed for various sectors in the Netherlands. In this early phase of development the main purpose is to learn. Later when monitoring and evaluation system is adopted and is frequently repeated, the wheels can also be used to compare sectors and account for progress. The first development showed that flood protection is the most advanced policy area within the Netherlands for adaptation. Relevant targets have been set in the Delta decisions, policies and measures are defined to achieve the targets. Also some form of monitoring has been set-up even for the output of the policy process. In other sectors like health and nature conservation, climate adaptation policies are seldom explicitly developed. But even in most of these other sectors much is going on already, although seldom assigned an adaptation policy.

The monitoring wheel for flood protection as an example to show the progress and level of implementation in adaptation policies specific sectors in the Netherlands. The inner circle includes five steps of the policy cycle (linked to knowing, wanting & working concept of the Dutch NAS), the out circle depicts the state of play of 16 underlying performance areas. Signposts are important in two stages in the cycle.

More info:

van Minnen et al. (2015) http://www.pbl.nl/sites/default/files/cms/publicaties/PBL-2015-Ontwerp-voor-een-nationale-adaptatiemonitor-1640.pdf   (only in Dutch)

Note: English version of this diagram will be included in the final draft

comments (2)

In Switzerland a Theory of Change approach is not used, however there are strong similarities in the way an Impact Model forms the basis of the MRE system (see box 2.13). This model places emphasis on understand the complex relationships within the adaptation process and on examining the expected, and actual, causal links from concept to impacts. This reveals the assumptions made during the development of the NAS which can then be tested using the Impact Model approach.

comments (0)

Box 2.13 Switzerland – Forming the basis of a national MRE system with the use of an Impact Model (rough draft below):

  • An Impact Model is a simple and efficient tool to demonstrate, communicate and facilitate discussion on the complex relationships related to climate change adaptation (CCA)
  • Indicators can provide relevant information on risk and vulnerability but less on the effectiveness of adaptation measures
  • The key challenges for the Swiss MRE system were a) setting objectives and thresholds for evaluating adaptation b) capturing the causality between the expected and the actual outcome of an adaptation measure and c) the short timeframe between the adoption and the evaluation of the action plan.

The Swiss Adaptation strategy was developed as framework for coordination at federal level. In the first part, the goals, challenges and fields of action for adapting to climate change are described. The second part is an action plan with 63 adaptation measures. Following the adoption of the action plan, the Swiss Federal Office for the Environment (FOEN) was mandated to report on the progress made and the effectiveness achieved in adaptation to the Federal Council by the end of 2017.  In response to this mandate the FOEN is setting up the framework for the development of an implementable, user-friendly and meaningful monitoring and evaluation (MRE) system for tracking climate change adaptation in Switzerland.

An Impact Model (see figure 2.4) forms the basis of the Swiss national MRE system. The model consists of five evaluation ‘objects’; concept; implementation; output; outcome and  impact, and sets out the logic underpinning the flow from one object to another.

Figure 2.4: Impact model (awaiting updated figure from Switzerland in September, version below is a place-holder)

FOEN is planning to specify an “Impact Model” for each risk. This will take into account the 12 main risks identified in the Swiss NAS as well as the adaptation measures specified in the action plan to respond to these risks. Initially, indicators were to be developed to evaluate effectiveness, however, it was concluded that while they may mainly provide important information on the vulnerability or risks, indicators are less informative when it comes to assessing effectiveness.


The impact model is a simple and effective tool to demonstrate, communicate and facilitate discussion on the complex relationships related to climate change adaptation (CCA). It has helped to optimise the use of existing knowledge and experience of stakeholders within Switzerland and has supported learning. However, a number of challenges have been identified including setting objectives and thresholds for evaluating adaptation; capturing the causality between the expected and the actual outcome of an adaptation measure; and the short timeframe between the adoption and the evaluation of the action plan.

Link (German, French, Italian): http://www.bafu.admin.ch/klima/13877/14401/14899/index.html?lang=de

comments (2)