Background

In the first years of CLIMAS, we learned a lot from many stakeholders about their uses of and needs for hydroclimatic information and forecasts. We found that decision makers had many reasons for not using seasonal climate forecasts. Some of the reasons included:

• The forecasts were for the wrong variables, were not issued near the time when they had to make decisions, or did not cover the specific region they were concerned with;
• The forecasts were difficult to interpret correctly, appeared too complicated, or were too vague;
• It was difficult to relate the forecasts to historic or recent conditions without a lot of extra effort;
• The decision makers couldn’t match which decisions could benefit from the forecasts because no one knew how the forecasts had performed for the situations important to them;
• The decision makers simply didn’t have the flexibility to take advantage of the forecasts, sometimes because of institutional limitations, and other times due to lack of resources or alternatives.

Some barriers to using forecasts may require many years to overcome (e.g., developing forecast models for new variables and changing institutional procedures and policies). We focused our research efforts on forecast assessment, to help decision makers better decide whether forecasts were appropriate for their specific applications. Forecast assessment included 1) research on forecast communication and interpretation, 2) relating forecasts to ancillary information (e.g., historical and recent observations), and 3) evaluating forecast performance using methods that reflect the many different ways that forecasts may convey information.

Decision Support Tools

Overview

Based on stakeholder interest in evaluation and verification of forecasts, we have devoted significant effort to the development of an online interactive Forecast Evaluation Tool (FET). Earlier stakeholder interactions made clear that long-term acceptance of a web tool by resource managers as a useful system to support decision making required significant commitment to development of a “commercial quality” website. To address this issue, we began a partnership with the Hydrologic Data and Information System (HyDIS) project now at UC Irvine as the Center for Hydrometeorology and Remote Sensing (CHRS), and a project under the GEWEX America Prediction Project (GAPP). Coordination with these projects brought into play a software engineering approach to achieve reliable support of multiple simultaneous users on a variety of browsers, as well as an expert web programming team to develop Java-based tools and implement a system for managing incremental website enhancements. We have now developed an Internet-based interactive Forecast Evaluation Tool that decision makers can customize to their specific interests. The ongoing decision-support tool project aims to improve the capabilities of the forecast evaluation tool and to develop decision-support capabilities for other forms of hydroclimatic information.

Methods

We develop improved decision-support tools using programming methods and software that meet industry standards. We are committed to open-source program code. Our methods allow the FET and other decision-support tools to be transferable to other regions and entities, and scalable to larger and smaller spatial domains, and greater numbers of users. We have used guided interactive workshops, key informant interviews, and structured survey instruments to garner information on the usefulness and usability of forecast evaluation formats and the FET website.

Results

We developed the Forecast Evaluation Tool. Based on feedback from workshops and other formal and informal interactions with website users, we've added features to the FET, including:

• A tutorial on correct interpretation of probabilistic seasonal climate outlooks;
• A feature that allows forecast comparisons with observed climate data;
• A feature to explore historical data and select segments of those data as analogs of possible future conditions;
• The modified Heidke Skill Score as a measure for evaluating forecast performance;
• Evaluation of seasonal climate outlooks for Alaska;
• Evaluation of NOAA-NWS downscaled local temperature outlooks.

The FET is now used by the NOAA-National Weather Service Climate Services Division to train NWS climate focal point personnel. Additionally, our forecast assessment web tools are designed to help decision makers get the most out of a variety of different forecasts. One such tool, the Climate Information Delivery and Decision Support System (CLIDDSS), provides information and decision support based off a paradigm of systematically providing improved forecast and information products. These products support the broadest range of decisions in an equitable manner, such as accommodating different levels of technical abilities. CLIDDSS allows users to:

• Use project folders to store selected products for multiple applications and efficient access over repeated site visits;
• Share their product suites with identified groups for discussion and consensus interpretation;
• Generate PDF reports that ensure inclusion of ancillary information combined with the ability to add customized interpretive comments (e.g., for delivery of information by intermediaries such as extension agents or state climatologists).

CLIDDSS also tracks web tool usage to provide ongoing feedback to operational agencies, science managers, and researchers about which products are preferred by various types of users and applications. CLIDDSS was designed explicitly for transfer to operations and to be scalable to serve intense usage such as hundreds of users at once.

Ensemble Water Supply Outlooks

Overview

As water demand increases and water management operations become more constrained, forecast users will require more and better information about the uncertainty associated with the forecasted streamflow. Current water supply forecasts for the western United States are based largely on statistical regression equations that are developed mostly from monthly precipitation, recent snow-water equivalent, and past streamflow observations. These forecasts lack information about the uncertainty of the predictions, which limits the potential for large improvements in forecast accuracy.

To provide an objective means with which to generate streamflow forecasts with uncertainty, the National Weather Service (NWS) Ensemble Streamflow Prediction (ESP) method was developed. The ESP system uses conceptual hydrologic models and historical data to generate a set, or ensemble, of possible streamflow scenarios conditioned on the initial states of a given basin.

The purpose of the ensemble streamflow predictions evaluations for water supply forecasting was to examine the applicability of traditional statistical techniques and distribution-oriented measures for the evaluation of ESP forecasts, and to give insight into potential operational forecast performance based on simulated historical ESP forecasts for the Colorado River basin.

Methods

ESP forecasts encompass the uncertainty in the hydrologic models, watershed conditions, future meteorology, and streamflow estimates. Information about the state of the watershed (e.g., snowpack water content) at the beginning of the forecast period is used to start the hydrologic model. ESP then uses past meteorology events as possible representations of future events. Thus, historical temperature and precipitation data are used as hydrologic model input to generate streamflow estimates for the future forecast period. Each set of meteorologic data from the different years, and spanning the forecast period, produces a single streamflow trace. The resulting ensemble of traces is used to create a probability distribution of streamflows that constitutes the forecast.

In cooperation with the Colorado Basin River Forecast Center (CBRFC), we evaluated their experimental ESP forecasts for 14 locations in the Colorado River Basin where flows have not been affected by regulations or diversions. ESP output has not been systematically archived until recently, making forecast evaluation difficult. Thus we used the NWS Ensemble Streamflow Prediction Verification System to simulate (hindcast) water supply outlooks for the 14 study locations discussed above. A variety of statistics for measuring forecast quality were used to evaluate the probabilistic forecasts, including mean absolute error, ranked probability score, reliability, and discrimination.

Results

We evaluated the ESP hindcasts using a mix of verification criteria, including traditional statistics and distribution-oriented measures. Selected key results from our study:

• Examination of specific ensemble traces (e.g., minimum-error and historical traces) can provide insight about the limitations of the forecast system and process, including proper model identification and parameterization, and the respective roles of initial conditions and meteorological uncertainty in affecting basin response.
• Headwater locations showed different forecast performance behavior across the upper and lower Colorado basins, but common behavior within the basins. Overall, forecasts are better for locations in the upper basin, and forecasts issued March 15 and later are generally the best.
• We recommend that forecasting agencies issue probabilistic forecast products that describe the entire forecasted distribution, or at least several portions of the distribution that have meaning for practical applications.
• Because probabilistic forecasts and their verification criteria are different in character than traditional forecast products, we recommend education efforts focused on the proper interpretation, evaluation, and application of new products.

Historical Water Supply Outlooks

Overview

Official US government seasonal streamflow forecasts (called water supply outlooks) are issued jointly by the National Weather Service (NWS) and the US Department of Agriculture (USDA) Natural Resources Conservation Service (NRCS). In Arizona, the Salt River Project (water and energy provider for the Phoenix metropolitan area) also participates. The period for which forecasts are made has varied over time, but now is generally April–July in the Upper Colorado River Basin and January–May in the Lower Colorado River Basin.

This project aims to establish a quantitative baseline of forecast performance by conducting the most comprehensive evaluation of official historical forecasts ever attempted for the Colorado River Basin.

Methods

We compared forecasts with “observed” values at 55 streamflow locations in the Colorado River Basin. The forecast flows represent the volume of water that would have occurred in the absence of diversions or regulations (such as by dams, irrigation, or municipal use). The actual “observed” flows are combined with estimates of withdrawals and diversions to reconstruct what the natural flows at each site would have been in the absence of these activities.

At each site, the observed flows were used to determine three flow levels for each forecast period: low flows (the lowest 30 percent of observations), moderate flows (the middle 40 percent), and high flows (the highest 30 percent of observations).

Several different statistical tests were used to examine the quality of the historical forecasts. These included basic statistical tests, such as root-mean-square error and correlation; categorical tests, such as the probability of detection and false alarm rate; probabilistic tests, such as Brier scores and the ranked probability score; and distributive statistics, such as reliability and discrimination.

Results

The study revealed that predictions of flows on the majority of streams have been very conservative. Below-average flows are often over-predicted (forecast values are too high) and above-average flows are under-predicted (forecast values are too low). This problem is most pronounced for early forecasts (i.e. January) at many locations, but improves with later forecasts (i.e. May).

For the low and high flows there was a low false alarm rate, which means that when low and high flows are forecast, those forecasts are generally accurate, and such flows do occur. However, for low and high flows there was also a low probability of detection at most sites—in other words, low and high flows actually occurred far more often than they were forecasted. Moderate flows, on the other hand, had a very high probability of detection, but also a very high false alarm rate, indicating that moderate flows are forecasted more frequently than they actually occur.

There was good discrimination between high and low flows, particularly with forecasts issued later in the year. This means that when high flows were forecasted, low flows rarely occurred, and vice versa. The accuracy of forecasts tended to improve with each month, so that forecasts issued in May tended to be much more reliable than those issued in January.

Seasonal Climate Outlooks

Overview

The purpose of the seasonal climate outlook evaluation project was to develop multiple evaluation criteria, in order to meet the needs of a wide variety of forecast users and a broad spectrum of user technical sophistication, to examine stakeholders’ forecast needs and the potential application of forecast skill to guide forecast use, and to develop guidelines for the effective communication of seasonal climate outlooks.

Methods

The Climate Prediction Center (CPC) outlooks predict the probability of seasonal average temperatures or seasonal total precipitation falling into one of three categories or terciles (warm, neutral, and cool for temperature and wet, neutral, and dry for precipitation), based on the distribution of observed data from 1971–2000. We evaluated forecast skill using:

• Categorical measures, such as probability of detection and false-alarm rate, which track how often forecasts are correct or incorrect, respectively;
• Brier scores, which consider the strength of the probability given to a specific category (e.g., warm or cold);
• Multi-category scores, such as ranked probability skill score, which evaluate the full range of conditions;
• Conditional distribution diagrams, such as discrimination and reliability diagrams, which can identify forecast skill for specific situations.

We also developed a set of diagrams to show CPC seasonal outlooks and their skill for all lead times leading up to a particular target forecast date.

We conducted extensive interviews and convened small workshops, in order to examine stakeholders’ needs and to learn about impediments to forecast communication.

Results

The results of our study have been published in a 2002 article in the Bulletin of the American Meteorological Society.  Some key findings with regard to the Southwest include the following:

• Forecast users frequently interpret the probabilities shown on NOAA CPC outlook maps as deterministic quantities. Consequently, great care must be used when communicating probabilistic forecasts to users;
• NOAA CPC seasonal temperature outlooks exhibit considerable skill in the Southwest;
• NOAA CPC seasonal precipitation outlooks are skillful in the Southwest for the winter months, but show almost no skill during the summer months. In fact, very few forecasts are issued for summer precipitation in the Southwest;
• From the perspective of water managers in the Southwest, winter precipitation outlooks made during the fall and winter are better than climatology according to all criteria. However, forecast skill is poor in the upper Colorado River basin—the source of the majority of the runoff into the Colorado River;
• Climate forecasts for the Southwest perform better from the perspective of cattle ranchers making use of winter range, given the greater climate predictability during winter;
• To the extent that fire risks are affected by winter conditions, forecast skill even at long lead times may prove useful to fire managers.

Related Publications

Franz, K., H. Hartmann, S. Sorooshian, and R. Bales. 2003. An evaluation of National Weather Service ensemble streamflow predictions for water supply forecasting in the Colorado River Basin. Journal of Hydrometeorology, 4:1105-1118.

Hartmann, H. 2005. Use of climate information in water resources management. In: Encyclopedia of Hydrological Sciences, M.G. Anderson (Ed.), John Wiley Sons Ltd., West Sussex, UK, Chapter 202.

Hartmann, H., R. Bales, and S. Sorooshian. 2002. Weather, climate, and hydrologic forecasting for the U.S. Southwest: a survey. Climate Research, 21:239-258.

Hartmann, H., A. Bradley, and A. Hamlet. 2003. Advanced hydrologic prediction for improving water management. In: Lawford, R., Fort, D., Hartmann, H. C., and S. Eden (Eds.), Water: Science, Policy, and Management. Water Resources Monograph 16, American Geophysical Union, Washington, DC, pp.285-307.

Hartmann, H., T. Pagano, S. Sorooshian, and R. Bales. 2002. Confidence builders: evaluating seasonal climate forecasts from user perspectives. Bulletin of the American Meteorological Society, 83(5):683-698.

Pagano, T., H. Hartmann, and S. Sorooshian. 2002. The role and usability of climate forecasts for water management in Arizona: a case study of the 1997-98 El Niño. Climate Research, 21:259-269.

Pagano, T., H. Hartmann, and S. Sorooshian. 2001. Use of climate forecasts for water management: Arizona and the 1997-98 El Niño. Journal of the American Water Resources Association, 37(5):1139-1152.