From the ABC 7 Weather team

UCACN Responds To Cliff Mass' Blog On U.S. Forecast Accuracy

April 13, 2012 - 05:00 AM
Text size Decrease Increase

A response to recent blog by Cliff mass about the accuracy of U.S. forecasts

This guest blog is a response by the UCAR Community Advisory Committee to NCEP  to a recent blog by Dr. Cliff Mass on the state of numerical weather forecasting in the U.S.  There are important discussions going on within the meteorological community (both science and budgets) which impact the basic "superstructure" of  "NWP" the core of weather forecasting. I think this response is well worth reading- Bob Ryan.  The views expressed do not necessarily reflect those of ABC7 or NC8.

As noted in Prof. Cliff Mass’s blog postings on 18 March and 7 April 2012, the performance measures for global model-based guidance for U.S. weather forecasts show the National Centers for Environmental Prediction (NCEP) lagging behind the European Centre for Medium-range Weather Forecasts (ECMWF) and the Meteorological Office of the United Kingdom (UKMO). The blog provided a number of sharp criticisms of NCEP, the National Weather Service (NWS) and the National Oceanic and Atmospheric Administration (NOAA), and it noted that, “a blue ribbon panel did a review of NCEP in 2009 and came to similar conclusions.” We, the members of the UCAR Community Advisory Committee for NCEP (UCACN), have followed up on that review and we find, in contrast to the statements in the blog, that NCEP is a significantly different place than it was even a few years ago with considerable promise for the future. We are writing to comment on some of Dr. Mass’s main points.

An independent review committee was commissioned by NCEP in 2009, managed through the University Corporation for Atmospheric Research (UCAR), and charged with critically reviewing the nine Centers that compose NCEP as well as the office of the Director of NCEP. This blog provided a link to one of the ten reports (EMC; information about the review and the other 9 reports can be found here: The UCAR review committee found many gaps and weaknesses (as well as strengths) in various parts of the NCEP enterprise, as suggested in the blog, and it made over 260 specific recommendations to help NCEP improve its products and services. One of the major recommendations was to form an external advisory committee to receive more frequent and expert advice from stakeholders in the academic and business communities. NCEP has embraced that recommendation and sought help from UCAR, which formed the UCAR Community Advisory Committee for NCEP (UCACN). Fred Carr (University of Oklahoma) and Jim Kinter (Center for Ocean-Land-Atmosphere Studies – COLA and George Mason University - GMU) were asked to serve as founding co-chairs of UCACN.
The UCACN has met once (in October 2011) since being formed, and the report of that meeting is also on the above web site. Based on the 2009 NCEP Review and the findings of the UCACN committee, we have the following comments on the 5 major points raised in the March 18 blog entry, repeated for convenience here:

1. The U.S. has inadequate computer power available for numerical weather prediction.
This is more correctly stated as NCEP has far less computer power available for its short and medium-range (0-16 days) forecast products (single plus ensemble runs) than does the ECMWF (and other centers). As rightly noted in Part II of Prof. Mass’s blog, two reasons for this are that (1) the NCEP supercomputer has less power than its competitors, and (2) NCEP's mission is an order of magnitude broader than ECMWF's, encompassing the prediction of not only medium-range weather but also seasonal and annual climate, marine and aviation weather, ocean waves, currents and storm surges, space weather, severe storms, hurricanes, air quality, etc. with mandates for ecosystem and decadal forecasting in the future. NCEP is under-resourced for this broad mission, a finding that motivated the 2009 NCEP Review to recommend that the mission be adjusted to fit the resources or that more resources be found. However, it is impossible for NCEP to discontinue services mandated by higher authorities, and it is extremely difficult to increase resources when NWS and NOAA budgets are being cut. The blog is correct in pointing out that there is excellent justification for substantially higher computer capability for operational prediction; in fact, an increase by a factor of at least 10 was recommended in the 2009 NCEP Review.
To update the information provided in Part II of the blog on the topic of computer procurement, NOAA recently (Feb 2012) awarded a $500 million, 10-year contract for the next generation operational high performance computing (HPC) system to IBM. An IBM iDataPlex system, which is a 149 teraflops (TF) system with 7,168 cores, based on the Intel Sandy Bridge chip, is expected to be operational by the end of 2013. A larger and more capable system was originally proposed by NCEP/NWS and NOAA, but budget constraints at a higher level restricted the size and speed of the system purchased. While the new system is rated lower than the 150-TF system at the UKMO and the 182-TF system at ECMWF and will be considerably smaller than machines available to academic researchers in the U.S., it is a step up in capability and capacity for operational weather prediction.

Supercomputer Data Center

It should be noted that HPC requirements are not limited to the supercomputer, but also include concomitant storage capacity, network bandwidth, information technology security, and reliability. The numerical guidance for forecasts has a 99.99% reliability requirement that demands a fully-redundant backup facility, removed from the main facility, with sub-second failover capability. As a result, Prof. Mass’s estimate of the cost of a 100,000-core system at $25 million neglects the total cost of operational, time-sensitive environmental prediction faced by NCEP. It is also noted, in response to the blog’s “not forward-leaning” comment, that NCEP’s current HPC system is the same (though with less capacity) as the ECMWF one, and the 2013 system will be the same (likewise with much less capacity) as that recently purchased by the National Center for Atmospheric Research (NCAR); all these systems are bid competitively, so any vendor can participate. 

2. The U.S. has used inferior data assimilation.
This statement is correct, and the ECMWF analysis is better. However, as an example that partially refutes reason #3 below, NCEP is partnering with NASA, a NOAA research lab and the Univ. of Oklahoma to develop ensemble-based hybrid (meaning combined with 3D- and 4DVAR) assimilation systems that should help them improve forecast skill. The new system being tested now significantly improves the Global Forecast System (GFS) results and will be implemented in May 2012.

3. The NWS numerical weather prediction effort has been isolated and has not taken advantage of the research community.
This is a frequent criticism that is only partially true. Many of NCEP’s modeling systems have come from other institutions, and over 70% of the modeling codes currently running in the NCEP Production Suite originated outside of NCEP. For example, EMC is partnering with the Developmental Testbed Center (DTC) at NCAR to facilitate transition of advances in mesoscale modeling into operation. Also, EMC and NCEP leadership have recently initiated many collaborative activities both in the US and abroad. In particular, NCEP has joined with the Canadian Meteorological Center (CMC) to create the North American Ensemble Forecast System, and the NCEP Climate Forecast System is included in the EuroSIP multi-model ensemble to improve seasonal predictions.

NAEFS Model - North American Ensemble Forecast System

This is evidence that NCEP is heeding the research community’s finding that multi-model ensembles are advantageous for both weather and climate prediction.
However, the ability to have an effective Visiting Scientist Program at NCEP has indeed been limited by lack of space and funds. The late summer 2012 move to the new NOAA Center for Weather and Climate Prediction (NCWCP), located in a research park in College Park, MD, will significantly improve opportunities for NCEP to interact with the external research community. Within the facility there are 40 spaces identified for visiting researchers, and the UCACN and NCEP are currently working with partners to obtain adequate resources to support visiting scientists.

4. The NWS approach to weather related research has been ineffective and divided.
As the blog later indicated, this statement is more correct if the word NOAA is substituted for NWS. The research components of NOAA and the operational weather and climate prediction component (NWS/NCEP) are bureaucratically separated and it requires orders from the NOAA Administrator to overcome barriers between them. The 2011 UCACN Annual Report recommended that interactions between NOAA research labs and NCEP need to be much more effective. Also, while NOAA has a limited external grants program for climate, it allocates even less funding for weather and NWP research, thus limiting academic scientists from receiving funding to support research that might directly benefit NCEP. In spite of all these barriers, NCEP has initiated and benefitted from collaborations with sister agencies, and is trying hard to start additional partnerships. The programs that NOAA funds to support weather and climate research should be expanded and better focused on numerical modeling to address this need more directly.

5. Lack of leadership.
This particular criticism is certainly not true about NCEP leadership. The UCACN can unequivocally state that NCEP leaders are a highly dedicated group who are not satisfied with the status quo, and who are working as best they can with the resources they have to improve the situation. We note that NCEP leadership is tracking all 263 of the 2009 Review recommendations to their completion, and many improvements have already been made. We know that the NWS Director had to make many painful budget cuts this year, but the NWS protected an important $10M increment for the new computer procurement, and did not cut NCEP’s funding. The UCACN has briefed NOAA officials about how to improve US NWP, but NOAA is dealing with its own budget cuts.
NCEP is doing its best within limited resources and disadvantageous federal personnel policies. Clearly the priorities for NCEP are not being set appropriately by NOAA, the Department of Commence or Congress. Many in Congress will not support funding increases for any cause, no matter how economically beneficial. Since NOAA/NWS/NCEP employees are forbidden from lobbying on their agency’s behalf, it is up to private citizens and the industries that rely on weather and seasonal forecasts to advance the cause for improved NWP.
As an aside, we note that NCEP has to resource and manage more than just computers and models. It also directs seven forecast centers, such as the National Hurricane Center, the Storm Prediction Center, etc., most of which are the best in the world in their respective missions. It is not sufficient to consider a single score of model forecast performance as indicative of the quality of the entire organization.
There are many challenges facing the U.S. weather prediction effort, and while there remain systemic impediments to making progress, the UCACN finds that NCEP is much more receptive to working with the external community. We expect significant improvement and accomplishment in the years to come.

Read More:
Short URL:
comments powered by Disqus