Planning for a new downstream demand-side management (DSM) program or pilot is always a challenge as there are so many unknowns. What interventions will convince customers to change their behavior to buying efficient products or implementing more efficient projects: incentives, on-bill financing, expert design guidance? How effective will those interventions be and how much will they cost? These challenges are only magnified when developing a midstream or upstream market transformation program because utilities have no direct interaction with the end user, which brings up a whole new set of important questions. How will retailers, manufacturers, and distributors respond to the program? How will customers respond to the changes in the market? What does that mean for changes in program costs and benefits over time? And how will all of these changes impact budgets and program cost-effectiveness? By foregoing direct interaction with customer, upstream and midstream energy efficiency programs offer energy efficiency program administrators an opportunity to achieve large program volume at low administrative cost by shifting the direct interaction of the program from a large number of end-users to a small number of influential market actors (retailers, distributors, or manufacturers). But to do so, they have to give up a measure of control over factors that, nonetheless, impact program success and cost-effectiveness.
Due to the complexity of these questions and the number of moving parts, program planners often try to assess midstream or upstream programs under a small set of scenarios (e.g., how cost effective would the program be if all costs come down rapidly and market uptake is high vs. all costs remain high and market uptake is low?). But what these scenarios gain in simplicity, they lack in nuance and depth. They don’t provide insight into more realistic scenarios in which some things go well, and others don’t, they don’t directly reflect what planners know about the underlying unknowns in the market, and they don’t accurately reflect how much that underlying unpredictability drives uncertainty in outcomes.
A POSSIBLE SOLUTION
Rather than limiting program design and planning to simplistic scenarios, a Monte Carlo simulation allows program planners to include a variety of realistic cases, showing a range possible outcomes of planning decisions and external factors, and to assess the impact of risk. The capacity of the simulation to account for so many variables allows for better decision-making under uncertainty.
A Monte Carlo simulation is a mathematical technique that was first developed in the 30s but not in common practice until recent advances in computing power. It allows analysts to account for a vast range of uncertainties through an iterative quantitative analysis process, using random draws from distributions of inputs, much like a gambling game at one of the famous casinos in Monte Carlo, Monaco. Here, instead of throwing the dice a few times to estimate how a program will react under different scenarios, modern computers allow Monte Carlo simulation to “throw the dice” thousands and thousands of times in order to build both a range of possible outcomes and even the probabilities that they will occur for any choice of action. This type of simulation provides a program designer with the outcomes of the most aggressive and conservative scenarios, along with a range of possible outcomes for everything in between, ensuring that program design decisions can take into account the range of possible outcomes and analyze the overall systemic risks and opportunities for their portfolios.
UNDER THE HOOD: HOW MONTE CARLO SIMULATIONS WORK
Monte Carlo simulations work by building ranges of possible outcomes through the substitution of values from a probability distribution for any specific component that has uncertainty, as well as sets of other fixed inputs. In the case of a midstream program design at a utility, these components could be sales volume, the dollar amount of a rebate, or the administrative cost of the program. A Monte Carlo simulation will sample values at random from a probability distribution over and over again, which will give each uncertain variable in a program design a range of different probabilities of different outcomes occurring. Depending on the number of uncertainties and the ranges specified for them, a Monte Carlo simulation could involve tens of thousands of iterations before it is complete, building a probability distribution of a range of possible outcomes along the way. This distribution provides a much more comprehensive view of what may happen when compared to traditional risk assessments, because it tells a program designer not only what could happen, but how likely it is to happen.
MONTE CARLO SIMULATIONS IN MIDSTREAM PROGRAM DESIGNS
Monte Carlo simulations are used in a broad range of industries, from astronomy to insurance, but can be particularly useful in program design. As an example, EMI Consulting recently implemented a Monte Carlo simulation to help a utility evaluate the prospects of cost-effectiveness for their Energy Star Retail Products Program (ESRPP). The ESRPP Program is a nationally-coordinated, midstream program design aimed at influencing retailers to alter their product assortment and to sell, promote, and demand more energy-efficient models of home appliances in specific product groups (e.g., clothes washers, dryers, and refrigerators). Utilities and other organizations (“Program Sponsors”) across the U.S. have partnered to develop and implement ESRPP. Each participating Program Sponsor pays participating retailers per-unit incentives for every program qualified unit they sell in each program category. The program theory holds that, by increasing the sales of energy-efficient models over less efficient models, the ESRPP Program will generate energy and demand savings for utility customers in the short-, mid-, and long-term through participating retailers, while also transforming the overall market towards higher efficiency in the long-term.
In this particular example, EMI Consulting was asked to provide a utility with key information to inform decisions about how to administer ESRPP, specifically by providing an assessment of the prospects for cost-effectiveness in each product group. The utility had developed seven possible scenarios, ranging from “conservative” to “aggressive” for certain key program elements, such as incremental measure cost, sales volume increases, and unit savings, in an Excel-based tool.
To help the utility develop a deeper understanding of the potential benefits of the program, EMI Consulting conducted a Monte Carlo simulation of the cost-effectiveness for each product group. To achieve this, EMI Consulting first replicated the utility’s cost-effectiveness calculations as an R-based tool, then simulated market outcomes by using variation in the sales data, the measurement uncertainty in sales increase rates, and empirically-based scenarios of program effects over time to conduct a simulation of the distribution of outcomes. The Monte Carlo analysis mimicked traditional analyses previously done by using cost-effectiveness assumptions generated by the utility, but it allowed for greater nuance in the scenarios by using random draws of sales volume simulations in each product category and by allowing program costs to vary at different rate to understand a full range of possible program variation.
By running over one million individual simulations across product groups, EMI Consulting was not only able to provide the utility with a single estimate of cost-effectiveness, we were able to provide both an average estimate of cost-effectiveness and information about how much uncertainty there was in that estimate so that the utility could understand the full range of outcomes and the risk associated with each product group. And because there were so many different inputs and scenarios, we also developed an R-based analysis tool to help the utility investigate the results and understand the distribution of outcomes for each scenario or set of scenarios.
This Monte Carlo simulation provided the utility client with the information required to meet a wide array of needs, including the ability to answer specific questions that executives would have about the impact of the program on the overall portfolio, based on real-world insight about the market. This type of approach could be useful any time there is uncertainty about how program and measure costs are going to change and when the extent of market uptake is unknown, such as launching new energy efficiency pilots or programs, or expanding electric vehicle charging infrastructure. For program evaluation, a Monte Carlo simulation could support the development and analysis of baseline market scenarios. Although the degree to which the results reflect reality is dependent on what is known about how the values will change, even coarse ranges of inputs can provide more meaningful bounds on the realm of possible outcomes than the simplistic approaches of the past that have lacked reasonable variation in important market characteristics.