SlideShare a Scribd company logo

From theory based policy evaluation to smart policy design: lessons learned from 20 cases studies on energy efficiency improvement

•
•
IEA DSM Implementing Agreement (IA)
IEA DSM Implementing Agreement (IA)

The document summarizes the findings of the AID-EE project, which evaluated 20 energy efficiency policies across Europe. Key lessons learned include: objectives should be specific, measurable, achievable, relevant and time-bound to guide policy; the impact of individual policies is difficult to isolate as most use packages with interacting instruments; and monitoring and evaluation have been a low priority, despite being important to determine policy effects and efficiency. A policy theory approach that defines clear objectives, indicators and expected relationships can help design, monitor and evaluate policies.

Related slideshows

Project monitoring and evaluation by Samuel Obino Mokaya
Project monitoring and evaluation by Samuel Obino MokayaProject monitoring and evaluation by Samuel Obino Mokaya
Project monitoring and evaluation by Samuel Obino Mokaya
 
Policy objectives
Policy objectivesPolicy objectives
Policy objectives
 
Training on Monitoring & Evaluation (M&E) of Adaptation and the NAP process
Training on Monitoring & Evaluation (M&E) of Adaptation and the NAP processTraining on Monitoring & Evaluation (M&E) of Adaptation and the NAP process
Training on Monitoring & Evaluation (M&E) of Adaptation and the NAP process
 
1 of 15
Download to read offline
From theory based policy evaluation to smart policy design: lessons learned from 20 case studies on energy efficiency improvement Results from the AID-EE project (Active Implementation of the proposed Directive on Energy Efficiency) Copenhagen April 19, 2006 Lars J. Nilsson, Lund University
AID-EE project Objectives 1. Reconstruct and analyze the policy implementation process with the purpose to identify and explain key factors behind success and failure of energy efficiency policies 2. Actively disseminate knowledge on implementation, monitoring and evaluation of policy instruments across Europe amongst others through a number of workshops
AID-EE project Work programme WP8: Common dissemination activities WP0:Projectmanagement WP1: Development of a common framework for assessment of policy instrument and selection of policy instruments WP2: Assessment of selected instruments in the households and service sector in the EU-25 WP3: Assessment of selected instruments in the industry sector in the EU-25 WP4: Assessment of instruments applied outside the EU-25, in other sectors or cross cutting instruments WP7: Dissemination actvities WP5: Overall impact assessment of current policies and potential 'best practice' policies WP6: Development of guidelines on monitoring and evaluation and synthesis of project results
AID-EE project Approach •Ex-post evaluation of 20 instruments applied in different sectors (households, services, industry, transport) across Europe •Application of a standardised evaluation methodology based on the ‘theory based policy evaluation’, which not only focuses on the final impact (energy savings) but also on intermediate indicators
AID-EE project Policy instruments included • Voluntary agreement (DK) • Audit programme (SF) • Industrial EE network (NO) • Energy concept for industrial branches (DE) • Eco-driving (NL) • ACEA agreement (EU) • FEMP (US) • Top runner approach (JP) • Advice service (DE/NRW) • Mandatory targets for network companies (BL) • Appliances labelling (NL) • Building code (NL) • KfW Building modernisation programme (DE) • EEC (UK) • Local energy advisors (SE) • Energy+ (EU) • AirCond-regulation (CH) • EE Procurement group (SE) • Energy manager obligation (IT) • EE investment deduction scheme (NL)
AID-EE project Practical approach: 6 steps 1. Characterization of the policy instrument 2. Draw up a policy theory and map out cause- impact relationships 3. Translate theory to concrete indicators 4. Draw up a flow chart 5. Verify (and adjust) policy theory 6. Collect information to draw-up indicators and draw conclusions
AID-EE project Relationship with other instruments Cause-Impact Relationship Indicators Success and Fail factors 1. Ministry of Trade (MTI) introduces the Energy Audit Programme (EAP). Motiva is the implementing organisation. 2a. Motiva carries out various activities to promote and support energy auditing. 2b. Promotion campaign to make the EAP known. Quality and type of promotion; awareness of EAP. Awareness of need of promotion; good relations with media. 2c. Training and authorisation of auditors. Quality of training; number of persons authorised; competence of auditors. Not main focus of EAP; auditors responsible for training. 2d. Development of energy audit models and auditor's tools. Quality, relevance and usefulness of models and tools. Flexible development; co- operation with stakeholders; role of Motiva. 3. Motiva introduces subsidies for energy audits. Voluntary Agreement scheme (introduced 1998). 4. Companies and organisations hire auditors and energy audits are carried out. Volume of audits. Connection to VA scheme; level of subsidies; active promotion. 5. Saving potentials are identified Savings potentials in companies and organisations. Comprehensive programme; careful monitoring. 6. Motiva controls the quality of energy audits Share of audits that are controlled; quality of energy audits. Tight quality control a conscious decision. ESCO programme (introduced 2000). 7. Companies and organiastions carry out energy saving measures based on the energy audits. Energy saving measures carried out; amount of energy saved. Saving measures are voluntary; economic (and other) incentives for target groups. Example: policy theory for the Finnish Energy Audit Program
AID-EE project Preliminary results from the AID-EE project
AID-EE project Lessons learned (1) •Smart objectives are good starting point for policy making but they are not always there: –Specified: be as concrete as possible, what is aimed for, who is targeted, what seems the most appropriate instrument or policy package to achieve the target –Measurable: objectives have to be measurable to determine whether results / effects have been achieved at a later stage –Acceptable: commitment within the target group facilitates policy implementation –Realistic: with respect to desired effect, available budget, the timeframe, a combination of these –Timeframe: It should be clear when the result / effect is to be achieved
AID-EE project Lessons learned (2) • In most cases with policy packages it is not possible to isolate the energy saving impact of a single policy instrument. Each instrument has its own specific role: – Creating awareness, setting a standard, showing opportunities, providing financial means etc. – This means that the objective / target of an instrument in the package is not necessarily an energy saving target. • The interaction of the policy instruments counts up to the total impact: – Would the impact still be the same when leaving out one of the instruments? • The interaction is dynamic and asks for proper instrument management: – e.g. due to increase of eco-tax the payback time of a measure declines which might lead to an increased amount of free riders in a subsidy scheme. – Be critical on the instrument portfolio
AID-EE project Lessons learned (3) •Monitoring information is often lacking or is insufficient to determine the effect and efficiency of an instrument –Monitoring does not have priority in the design phase –The need for monitoring is often not addressed (e.g. by setting an annual budget for monitoring) –Because objectives are often not clear and measurable, monitoring and evaluation becomes difficult. –An explicit policy theory would facilitate monitoring and evaluation.
AID-EE project policy formulation policy implementation policy in practice explicit program theory operational model (indicators) monitoring/ evaluation possible reformulation/ reorganization of policies policy implementation How policy theory can help to design: •Smart objectives •Smart monitoring protocols
AID-EE project Preliminary summary conclusions • Often multiple and mumbled objectives. • There are policy packages and interplay between instruments. • Monitoring and verification has been a relatively low priority. • Organization matters – a clear mission and proper funding seem important for success. • Acknowledge the different natures of different instruments. – The concept of smart objectives is relevant for all types of instruments, but should be applied differently – Different instruments ask for different monitoring protocols • One should take into account relevant differences in: – Behavioral, cultural, and socio-demographic conditions – Market structures and policy/regulatory style – Climatic & geographical conditions • A policy/program theory approach can guide policy design and subsequent monitoring and evaluation efforts.
AID-EE project E4SD reflections - AID-EE implications •The directive provides much needed impetus for running a tighter ship. •Resource acquisition or market transformation – will only hard kWh’s count towards target? –Bottom-up quantifiable energy savings from sales data, participation rates, engineering estimates, M&V, etc. –Top-down estimated effect of taxes or tariffs from price elasticities. –But what is the effect of building declarations, development of supporting tools, demonstrations, education, etc? •Develop ”approved methodologies” in dialogue with policy owners (cf. CDM Executive Board)?
AID-EE project Further information: Lund University, Sweden Lars J. Nilsson +46 46 222 4683 email: Lars_J.Nilsson@miljo.lth.se OR Project coordinator Ecofys BV, the Netherlands Contact person: Mirjam Harmelink +31 30 2808495, email: M.Harmelink@ecofys.nl Project website: www.aid-ee.org

More Related Content

From theory based policy evaluation to smart policy design: lessons learned from 20 cases studies on energy efficiency improvement

  • 1. From theory based policy evaluation to smart policy design: lessons learned from 20 case studies on energy efficiency improvement Results from the AID-EE project (Active Implementation of the proposed Directive on Energy Efficiency) Copenhagen April 19, 2006 Lars J. Nilsson, Lund University
  • 2. AID-EE project Objectives 1. Reconstruct and analyze the policy implementation process with the purpose to identify and explain key factors behind success and failure of energy efficiency policies 2. Actively disseminate knowledge on implementation, monitoring and evaluation of policy instruments across Europe amongst others through a number of workshops
  • 3. AID-EE project Work programme WP8: Common dissemination activities WP0:Projectmanagement WP1: Development of a common framework for assessment of policy instrument and selection of policy instruments WP2: Assessment of selected instruments in the households and service sector in the EU-25 WP3: Assessment of selected instruments in the industry sector in the EU-25 WP4: Assessment of instruments applied outside the EU-25, in other sectors or cross cutting instruments WP7: Dissemination actvities WP5: Overall impact assessment of current policies and potential 'best practice' policies WP6: Development of guidelines on monitoring and evaluation and synthesis of project results
  • 4. AID-EE project Approach •Ex-post evaluation of 20 instruments applied in different sectors (households, services, industry, transport) across Europe •Application of a standardised evaluation methodology based on the ‘theory based policy evaluation’, which not only focuses on the final impact (energy savings) but also on intermediate indicators
  • 5. AID-EE project Policy instruments included • Voluntary agreement (DK) • Audit programme (SF) • Industrial EE network (NO) • Energy concept for industrial branches (DE) • Eco-driving (NL) • ACEA agreement (EU) • FEMP (US) • Top runner approach (JP) • Advice service (DE/NRW) • Mandatory targets for network companies (BL) • Appliances labelling (NL) • Building code (NL) • KfW Building modernisation programme (DE) • EEC (UK) • Local energy advisors (SE) • Energy+ (EU) • AirCond-regulation (CH) • EE Procurement group (SE) • Energy manager obligation (IT) • EE investment deduction scheme (NL)
  • 6. AID-EE project Practical approach: 6 steps 1. Characterization of the policy instrument 2. Draw up a policy theory and map out cause- impact relationships 3. Translate theory to concrete indicators 4. Draw up a flow chart 5. Verify (and adjust) policy theory 6. Collect information to draw-up indicators and draw conclusions
  • 7. AID-EE project Relationship with other instruments Cause-Impact Relationship Indicators Success and Fail factors 1. Ministry of Trade (MTI) introduces the Energy Audit Programme (EAP). Motiva is the implementing organisation. 2a. Motiva carries out various activities to promote and support energy auditing. 2b. Promotion campaign to make the EAP known. Quality and type of promotion; awareness of EAP. Awareness of need of promotion; good relations with media. 2c. Training and authorisation of auditors. Quality of training; number of persons authorised; competence of auditors. Not main focus of EAP; auditors responsible for training. 2d. Development of energy audit models and auditor's tools. Quality, relevance and usefulness of models and tools. Flexible development; co- operation with stakeholders; role of Motiva. 3. Motiva introduces subsidies for energy audits. Voluntary Agreement scheme (introduced 1998). 4. Companies and organisations hire auditors and energy audits are carried out. Volume of audits. Connection to VA scheme; level of subsidies; active promotion. 5. Saving potentials are identified Savings potentials in companies and organisations. Comprehensive programme; careful monitoring. 6. Motiva controls the quality of energy audits Share of audits that are controlled; quality of energy audits. Tight quality control a conscious decision. ESCO programme (introduced 2000). 7. Companies and organiastions carry out energy saving measures based on the energy audits. Energy saving measures carried out; amount of energy saved. Saving measures are voluntary; economic (and other) incentives for target groups. Example: policy theory for the Finnish Energy Audit Program
  • 8. AID-EE project Preliminary results from the AID-EE project
  • 9. AID-EE project Lessons learned (1) •Smart objectives are good starting point for policy making but they are not always there: –Specified: be as concrete as possible, what is aimed for, who is targeted, what seems the most appropriate instrument or policy package to achieve the target –Measurable: objectives have to be measurable to determine whether results / effects have been achieved at a later stage –Acceptable: commitment within the target group facilitates policy implementation –Realistic: with respect to desired effect, available budget, the timeframe, a combination of these –Timeframe: It should be clear when the result / effect is to be achieved
  • 10. AID-EE project Lessons learned (2) • In most cases with policy packages it is not possible to isolate the energy saving impact of a single policy instrument. Each instrument has its own specific role: – Creating awareness, setting a standard, showing opportunities, providing financial means etc. – This means that the objective / target of an instrument in the package is not necessarily an energy saving target. • The interaction of the policy instruments counts up to the total impact: – Would the impact still be the same when leaving out one of the instruments? • The interaction is dynamic and asks for proper instrument management: – e.g. due to increase of eco-tax the payback time of a measure declines which might lead to an increased amount of free riders in a subsidy scheme. – Be critical on the instrument portfolio
  • 11. AID-EE project Lessons learned (3) •Monitoring information is often lacking or is insufficient to determine the effect and efficiency of an instrument –Monitoring does not have priority in the design phase –The need for monitoring is often not addressed (e.g. by setting an annual budget for monitoring) –Because objectives are often not clear and measurable, monitoring and evaluation becomes difficult. –An explicit policy theory would facilitate monitoring and evaluation.
  • 12. AID-EE project policy formulation policy implementation policy in practice explicit program theory operational model (indicators) monitoring/ evaluation possible reformulation/ reorganization of policies policy implementation How policy theory can help to design: •Smart objectives •Smart monitoring protocols
  • 13. AID-EE project Preliminary summary conclusions • Often multiple and mumbled objectives. • There are policy packages and interplay between instruments. • Monitoring and verification has been a relatively low priority. • Organization matters – a clear mission and proper funding seem important for success. • Acknowledge the different natures of different instruments. – The concept of smart objectives is relevant for all types of instruments, but should be applied differently – Different instruments ask for different monitoring protocols • One should take into account relevant differences in: – Behavioral, cultural, and socio-demographic conditions – Market structures and policy/regulatory style – Climatic & geographical conditions • A policy/program theory approach can guide policy design and subsequent monitoring and evaluation efforts.
  • 14. AID-EE project E4SD reflections - AID-EE implications •The directive provides much needed impetus for running a tighter ship. •Resource acquisition or market transformation – will only hard kWh’s count towards target? –Bottom-up quantifiable energy savings from sales data, participation rates, engineering estimates, M&V, etc. –Top-down estimated effect of taxes or tariffs from price elasticities. –But what is the effect of building declarations, development of supporting tools, demonstrations, education, etc? •Develop ”approved methodologies” in dialogue with policy owners (cf. CDM Executive Board)?
  • 15. AID-EE project Further information: Lund University, Sweden Lars J. Nilsson +46 46 222 4683 email: Lars_J.Nilsson@miljo.lth.se OR Project coordinator Ecofys BV, the Netherlands Contact person: Mirjam Harmelink +31 30 2808495, email: M.Harmelink@ecofys.nl Project website: www.aid-ee.org