Mitigating Risk through Embedded Evaluation

By:

Utility-run energy efficiency programs continue to be a core part of utilities’ strategy to help lower customers’ energy costs, manage an increasingly complex grid, and curb carbon emissions. In 2018 alone, utilities across the United States spent $6.6 billion on energy efficiency programs. However, state-set savings goals for these programs continue to increase, while at the same time, traditional program channels are increasingly challenging due to changing baselines (e.g., residential lighting). Within this environment, program administrators are searching for methods that will mitigate the risks associated with program delivery and are increasingly turning to new methods of evaluation as part of their overall strategy. For example, in California, the investor-owned utilities are exploring the use of embedded evaluation as a method to mitigate risk associated with the delivery of energy efficiency programs.

However, what is “embedded evaluation” and “embedded M&V”? Unlike developmental evaluation or summative evaluation, it is not a formal academic evaluation framework. In the California context, embedded M&V means that energy efficiency programs should include the necessary systems to report out on program performance throughout program delivery (e.g., through ongoing analysis of participants’ billing data). In essence, this allows both implementation and administration staff to track the success of a program in near-real-time and in its progress towards a program’s stated goals. This is similar to the concept of “M&V 2.0” – using software solutions to continuously monitor the performance of programs through analysis of changes in energy consumption.

However, it’s important to remember that “E” in “EM&V.” Frequently, embedded evaluation focuses on tracking what is happening but loses focus on understanding why. At EMI Consulting, we believe evaluation results are not actionable if you only know what is happening. Knowing why provides a course of action, and any well-designed evaluation framework should be able to tell you both. As a result, our approach provides benefits associated with embedded evaluation, like frequent feedback on program performance, proactive risk identification. We track program performance through key indicators (e.g., project realization rates, customer satisfaction) and couple that with robust customer research to understand the responses of program participants and other market actors, allowing us to discover if and when a program is under-performing. We invest in up-front planning, stakeholder management, and consensus building to help build trust between program administrators and implementation staff. And we work collaboratively with program managers to develop recommendations that will truly be actionable and impactful on program performance. A couple of examples of EMI Consulting projects:

  • EMI Consulting recently developed an evaluation framework for an innovative targeted pilot program focused on GHG reductions. Importantly, our framework included both the methodology for quantifying the reductions in GHG on an on-going basis as well as the collection of qualitative data from pilot participants to better understand any observed trends in measured reductions. We sought to identify if there were particular approaches that were working well, or any barriers preventing participants from achieving greater reductions?
  • EMI Consulting has employed a comprehensive embedded evaluation approach for a custom program targeting large commercial and industrial customers. As part of this approach, our team works closely with the implementation contractor to review project applications and identify risks in the energy savings calculations from an evaluation perspective. We highlight factors like any assumptions that seem risky based on our past experience or if the engineering calculations align with best practices. While this review takes extra time, it has resulted in incredibly stable realization rates and increased participant satisfaction. In addition, by focusing on “why” the project might result in risk, the implementation contractor has been able to improve their systems over time.
  • Finally, to support the launch of an EV pilot, EMI Consulting developed a continuous improvement framework that would track key performance metrics critical to pilot success (e.g., number of chargers installed, average installation costs, participant satisfaction). An important component of this framework was assisting in the development of both applications and data integration tools to ensure that the pilot collected the appropriate data to inform these metrics. As part of the implementation of this framework, the pilot program team were able to increase their internal evaluation capabilities and track the performance of the pilot in real-time.

 

These examples highlight how embedded evaluation can support program delivery and mitigate the risks associated with innovative technologies or delivery mechanisms. However, an embedded evaluation framework alone is not enough to ensure that program portfolios succeed. Key to success is trust and collaboration. Our team places significant emphasis on building and maintaining relationships with all stakeholders to build trust with program staff which, in turn, makes our relationship more collaborative.

If you want to hear more about embedded evaluation, and how EMI Consulting can reduce risk for your programs, reach out to me and Quinn Parker at AESP’s National Conference and Nicole Thomas and Lisa Perry at CEDMC’S EM&V Forum. Or check out Katie Cary’s series on developmental evaluation.

And if you’d like to be part of the innovative team guiding our clients to a clean energy future, please check out the opportunities on our Careers page: (Senior Research Analyst, Research Analyst).