This project focused on forecast assessment to help decision makers better decide whether forecasts are appropriate for their specific applications. Forecast assessment included: a) research on forecast communication and interpretation; b) relating forecasts to ancillary information (e.g., historical and recent observations); and c) evaluating forecast performance using methods that reflect the many different ways that forecasts may convey information.
Results from this project helped inform the design of the online and interactive Forecast Evaluation Tool. This tool helps decision makers understand the accuracy and usefulness of CPC forecasts with several customizable analyses.
This project conducted forecast evaluations on four types of forecasts. Links to the following forecast evaluations provide a more complete picture of the specific evaluation, methods, and results.
In the first years of CLIMAS, we learned a lot from many stakeholders about their uses of and needs for hydroclimatic information and forecasts. We found that decision makers had many reasons for not using seasonal climate forecasts. Some of the reasons included:
Some barriers to using forecasts may require many years to overcome (e.g., developing forecast models for new variables and changing institutional procedures and policies). We focused our research efforts on forecast assessment, to help decision makers better decide whether forecasts were appropriate for their specific applications. Forecast assessment included 1) research on forecast communication and interpretation, 2) relating forecasts to ancillary information (e.g., historical and recent observations), and 3) evaluating forecast performance using methods that reflect the many different ways that forecasts may convey information.
Based on stakeholder interest in evaluation and verification of forecasts, we have devoted significant effort to the development of an online interactive Forecast Evaluation Tool (FET). Earlier stakeholder interactions made clear that long-term acceptance of a web tool by resource managers as a useful system to support decision making required significant commitment to development of a “commercial quality” website. To address this issue, we began a partnership with the Hydrologic Data and Information System (HyDIS) project now at UC Irvine as the Center for Hydrometeorology and Remote Sensing (CHRS), and a project under the GEWEX America Prediction Project (GAPP). Coordination with these projects brought into play a software engineering approach to achieve reliable support of multiple simultaneous users on a variety of browsers, as well as an expert web programming team to develop Java-based tools and implement a system for managing incremental website enhancements. We have now developed an Internet-based interactive Forecast Evaluation Tool that decision makers can customize to their specific interests. The ongoing decision-support tool project aims to improve the capabilities of the forecast evaluation tool and to develop decision-support capabilities for other forms of hydroclimatic information.
We develop improved decision-support tools using programming methods and software that meet industry standards. We are committed to open-source program code. Our methods allow the FET and other decision-support tools to be transferable to other regions and entities, and scalable to larger and smaller spatial domains, and greater numbers of users. We have used guided interactive workshops, key informant interviews, and structured survey instruments to garner information on the usefulness and usability of forecast evaluation formats and the FET website.
We developed the Forecast Evaluation Tool. Based on feedback from workshops and other formal and informal interactions with website users, we've added features to the FET, including:
The FET is now used by the NOAA-National Weather Service Climate Services Division to train NWS climate focal point personnel. Additionally, our forecast assessment web tools are designed to help decision makers get the most out of a variety of different forecasts. One such tool, the Climate Information Delivery and Decision Support System (CLIDDSS), provides information and decision support based off a paradigm of systematically providing improved forecast and information products. These products support the broadest range of decisions in an equitable manner, such as accommodating different levels of technical abilities. CLIDDSS allows users to:
CLIDDSS also tracks web tool usage to provide ongoing feedback to operational agencies, science managers, and researchers about which products are preferred by various types of users and applications. CLIDDSS was designed explicitly for transfer to operations and to be scalable to serve intense usage such as hundreds of users at once.
As water demand increases and water management operations become more constrained, forecast users will require more and better information about the uncertainty associated with the forecasted streamflow. Current water supply forecasts for the western United States are based largely on statistical regression equations that are developed mostly from monthly precipitation, recent snow-water equivalent, and past streamflow observations. These forecasts lack information about the uncertainty of the predictions, which limits the potential for large improvements in forecast accuracy.
To provide an objective means with which to generate streamflow forecasts with uncertainty, the National Weather Service (NWS) Ensemble Streamflow Prediction (ESP) method was developed. The ESP system uses conceptual hydrologic models and historical data to generate a set, or ensemble, of possible streamflow scenarios conditioned on the initial states of a given basin.
The purpose of the ensemble streamflow predictions evaluations for water supply forecasting was to examine the applicability of traditional statistical techniques and distribution-oriented measures for the evaluation of ESP forecasts, and to give insight into potential operational forecast performance based on simulated historical ESP forecasts for the Colorado River basin.
ESP forecasts encompass the uncertainty in the hydrologic models, watershed conditions, future meteorology, and streamflow estimates. Information about the state of the watershed (e.g., snowpack water content) at the beginning of the forecast period is used to start the hydrologic model. ESP then uses past meteorology events as possible representations of future events. Thus, historical temperature and precipitation data are used as hydrologic model input to generate streamflow estimates for the future forecast period. Each set of meteorologic data from the different years, and spanning the forecast period, produces a single streamflow trace. The resulting ensemble of traces is used to create a probability distribution of streamflows that constitutes the forecast.
In cooperation with the Colorado Basin River Forecast Center (CBRFC), we evaluated their experimental ESP forecasts for 14 locations in the Colorado River Basin where flows have not been affected by regulations or diversions. ESP output has not been systematically archived until recently, making forecast evaluation difficult. Thus we used the NWS Ensemble Streamflow Prediction Verification System to simulate (hindcast) water supply outlooks for the 14 study locations discussed above. A variety of statistics for measuring forecast quality were used to evaluate the probabilistic forecasts, including mean absolute error, ranked probability score, reliability, and discrimination.
We evaluated the ESP hindcasts using a mix of verification criteria, including traditional statistics and distribution-oriented measures. Selected key results from our study:
Official US government seasonal streamflow forecasts (called water supply outlooks) are issued jointly by the National Weather Service (NWS) and the US Department of Agriculture (USDA) Natural Resources Conservation Service (NRCS). In Arizona, the Salt River Project (water and energy provider for the Phoenix metropolitan area) also participates. The period for which forecasts are made has varied over time, but now is generally April–July in the Upper Colorado River Basin and January–May in the Lower Colorado River Basin.
This project aims to establish a quantitative baseline of forecast performance by conducting the most comprehensive evaluation of official historical forecasts ever attempted for the Colorado River Basin.
We compared forecasts with “observed” values at 55 streamflow locations in the Colorado River Basin. The forecast flows represent the volume of water that would have occurred in the absence of diversions or regulations (such as by dams, irrigation, or municipal use). The actual “observed” flows are combined with estimates of withdrawals and diversions to reconstruct what the natural flows at each site would have been in the absence of these activities.
At each site, the observed flows were used to determine three flow levels for each forecast period: low flows (the lowest 30 percent of observations), moderate flows (the middle 40 percent), and high flows (the highest 30 percent of observations).
Several different statistical tests were used to examine the quality of the historical forecasts. These included basic statistical tests, such as root-mean-square error and correlation; categorical tests, such as the probability of detection and false alarm rate; probabilistic tests, such as Brier scores and the ranked probability score; and distributive statistics, such as reliability and discrimination.
The study revealed that predictions of flows on the majority of streams have been very conservative. Below-average flows are often over-predicted (forecast values are too high) and above-average flows are under-predicted (forecast values are too low). This problem is most pronounced for early forecasts (i.e. January) at many locations, but improves with later forecasts (i.e. May).
For the low and high flows there was a low false alarm rate, which means that when low and high flows are forecast, those forecasts are generally accurate, and such flows do occur. However, for low and high flows there was also a low probability of detection at most sites—in other words, low and high flows actually occurred far more often than they were forecasted. Moderate flows, on the other hand, had a very high probability of detection, but also a very high false alarm rate, indicating that moderate flows are forecasted more frequently than they actually occur.
There was good discrimination between high and low flows, particularly with forecasts issued later in the year. This means that when high flows were forecasted, low flows rarely occurred, and vice versa. The accuracy of forecasts tended to improve with each month, so that forecasts issued in May tended to be much more reliable than those issued in January.
The purpose of the seasonal climate outlook evaluation project was to develop multiple evaluation criteria, in order to meet the needs of a wide variety of forecast users and a broad spectrum of user technical sophistication, to examine stakeholders’ forecast needs and the potential application of forecast skill to guide forecast use, and to develop guidelines for the effective communication of seasonal climate outlooks.
The Climate Prediction Center (CPC) outlooks predict the probability of seasonal average temperatures or seasonal total precipitation falling into one of three categories or terciles (warm, neutral, and cool for temperature and wet, neutral, and dry for precipitation), based on the distribution of observed data from 1971–2000. We evaluated forecast skill using:
We also developed a set of diagrams to show CPC seasonal outlooks and their skill for all lead times leading up to a particular target forecast date.
We conducted extensive interviews and convened small workshops, in order to examine stakeholders’ needs and to learn about impediments to forecast communication.
The results of our study have been published in a 2002 article in the Bulletin of the American Meteorological Society. Some key findings with regard to the Southwest include the following:
Franz, K., H. Hartmann, S. Sorooshian, and R. Bales. 2003. An evaluation of National Weather Service ensemble streamflow predictions for water supply forecasting in the Colorado River Basin. Journal of Hydrometeorology, 4:1105-1118.
Hartmann, H. 2005. Use of climate information in water resources management. In: Encyclopedia of Hydrological Sciences, M.G. Anderson (Ed.), John Wiley Sons Ltd., West Sussex, UK, Chapter 202.
Hartmann, H., R. Bales, and S. Sorooshian. 2002. Weather, climate, and hydrologic forecasting for the U.S. Southwest: a survey. Climate Research, 21:239-258.
Hartmann, H., A. Bradley, and A. Hamlet. 2003. Advanced hydrologic prediction for improving water management. In: Lawford, R., Fort, D., Hartmann, H. C., and S. Eden (Eds.), Water: Science, Policy, and Management. Water Resources Monograph 16, American Geophysical Union, Washington, DC, pp.285-307.
Hartmann, H., T. Pagano, S. Sorooshian, and R. Bales. 2002. Confidence builders: evaluating seasonal climate forecasts from user perspectives. Bulletin of the American Meteorological Society, 83(5):683-698.
Pagano, T., H. Hartmann, and S. Sorooshian. 2002. The role and usability of climate forecasts for water management in Arizona: a case study of the 1997-98 El Niño. Climate Research, 21:259-269.
Pagano, T., H. Hartmann, and S. Sorooshian. 2001. Use of climate forecasts for water management: Arizona and the 1997-98 El Niño. Journal of the American Water Resources Association, 37(5):1139-1152.