With the Major League Baseball season having just begun, I’m reminded of the truism that the best teams win their divisions in the regular season, but the hot teams win in the post-season playoffs. Why the difference? The regular season is 162 games long, but the post-season consists of just a few brief 5-game and 7-game series. And because of the huge random element that pervades the sport, in a single game (or a short series), the best teams often lose, and the worst teams often win.
The numbers are striking, and bear repeating. In a typical year, the best teams lose 40 percent of their games, and the worst teams win 40 percent of theirs. In the extreme, one of the best Major League Baseball teams ever - the 1927 New York Yankees – lost 29 percent of their games; and one of the worst teams in history – the 1962 New York Mets – won 25 percent of theirs. On any given day, anything can happen. Uncertainty is a fundamental part of the game, and any analysis that fails to recognize this is not only incomplete, but fundamentally flawed.
The same is true of analyses of environmental policies. Uncertainty is an absolutely fundamental aspect of environmental problems and the policies that are employed to address those problems. Any analysis that fails to recognize this runs the risk not only of being incomplete, but misleading as well. Judson Jaffe, formerly at Analysis Group, and I documented this in a study published in Regulation and Governance.
To estimate proposed regulations’ benefits and costs, analysts frequently rely on inputs that are uncertain – sometimes substantially so. Such uncertainties in underlying inputs are propagated through analyses, leading to uncertainty in ultimate benefit and cost estimates, which constitute the core of a Regulatory Impact Analysis (RIA), required by Presidential Executive Order for all “economically significant” proposed Federal regulations.
Despite this uncertainty, the most prominently displayed results in RIAs are typically single, apparently precise point estimates of benefits, costs, and net benefits (benefits minus costs), masking uncertainties inherent in their calculation and possibly obscuring tradeoffs among competing policy options. Historically, efforts to address uncertainty in RIAs have been very limited, but guidance set forth in the U.S. Office of Management and Budget’s (OMB) Circular A‑4 on Regulatory Analysis has the potential to enhance the information provided in RIAs regarding uncertainty in benefit and cost estimates. Circular A‑4 requires the development of a formal quantitative assessment of uncertainty regarding a regulation’s economic impact if either annual benefits or costs are expected to reach $1 billion.
Over the years, formal quantitative uncertainty assessments — known as Monte Carlo analyses — have become common in a variety of fields, including engineering, finance, and a number of scientific disciplines, as well as in “sabermetrics” (quantitative, especially statistical analysis of professional baseball), but rarely have such methods been employed in RIAs.
The first step in a Monte Carlo analysis involves the development of probability distributions of uncertain inputs to an analysis. These probability distributions reflect the implications of uncertainty regarding an input for the range of its possible values and the likelihood that each value is the true value. Once probability distributions of inputs to a benefit‑cost analysis are established, a Monte Carlo analysis is used to simulate the probability distribution of the regulation’s net benefits by carrying out the calculation of benefits and costs thousands, or even millions, of times. With each iteration of the calculations, new values are randomly drawn from each input’s probability distribution and used in the benefit and/or cost calculations. Over the course of these iterations, the frequency with which any given value is drawn for a particular input is governed by that input’s probability distribution. Importantly, any correlations among individual items in the benefit and cost calculations are taken into account. The resulting set of net benefit estimates characterizes the complete probability distribution of net benefits.
Uncertainty is inevitable in estimates of environmental regulations’ economic impacts, and assessments of the extent and nature of such uncertainty provides important information for policymakers evaluating proposed regulations. Such information offers a context for interpreting benefit and cost estimates, and can lead to point estimates of regulations= benefits and costs that differ from what would be produced by purely deterministic analyses (that ignore uncertainty). In addition, these assessments can help establish priorities for research.
Due to the complexity of interactions among uncertainties in inputs to RIAs, an accurate assessment of uncertainty can be gained only through the use of formal quantitative methods, such as Monte Carlo analysis. Although these methods can offer significant insights, they require only limited additional effort relative to that already expended on RIAs. Much of the data required for these analyses are already obtained by EPA in their preparation of RIAs; and widely available software allows the execution of Monte Carlo analysis in common spreadsheet programs on a desktop computer. In a specific application in the Regulation and Governance study, Jaffe and I demonstrate the use and advantages of employing formal quantitative analysis of uncertainty in a review of EPA’s 2004 RIA for its Nonroad Diesel Rule.
Formal quantitative assessments of uncertainty can mark a truly significant step forward in enhancing regulatory analysis under Presidential Executive Orders. They have the potential to improve substantially our understanding of the impact of environmental regulations, and thereby to lead to more informed policymaking.
