Great forecast, but not great communication-Assessing Sandy: Warnings and the Weather Enterprise (UPDATED: Now two NWS reasons why the first team was scuttled).
From time to time we feature a "Guest Blog". Nate Johnson is not only a fellow meteorologist, but will be getting a graduate degree in communication. A rare combination. My physical science of pure meteorology has made amazing advances in my life, but the promise of the great advance of science is still unfulfilled if we do not effectively communicate the forecast so everyone makes the best weather related decision. The objective forecast of hurricane Sandy was a near perfect forecast of modern meteorology. The decision to not issue "hurricane warnings" for the East Coast, and what this meant for public decision making, is still being strongly debated within the meteorological community as you will read in Nate's blog first published here.
Assessing Sandy: Warnings and the Weather Enterprise (UPDATED: Now two NWS reasons why the first team was scuttled)
Eleven days ago, I recapped the situation surrounding Hurricane Sandy, including the short-lived spin-up of the National Weather Service (NWS) Service Assessment team that would have been charged with investigating how the various arms of the agency performed before, during, and after Sandy’s brush with North Carolina and direct assault on the northeastern US. At that time, the official word from the NWS was that a broader, multi-agency approach to the federal government’s preparation for and response to Sandy was possible, and that if that didn’t happen, the NWS would pursue an internal assessment “as it always has”.
This week marks one month since Sandy’s landfall (and one month of ongoing cleanup and recovery in some of the affected areas, to boot), and while the extended time doesn’t preclude gathering reliable data upon which the organization and the broader weather enterprise can draw conclusions, we aren’t gaining anything by sitting on the sidelines while the clock ticks. The termination of the original team, the lack of a new assessment effort, and the ongoing delay has caught the attention of Rep. Paul Broun (R-GA), chairman of the House Subcommittee on Investigations and Oversight, who wrote in a letter to NOAA Administrator Dr. Jane Lubchenco, “it is imperative that the NWS act quickly and decisively in performing an assessment.”
There are rumblings that a NOAA-level (NWS’ parent agency) review may be announced shortly – perhaps in response to Rep. Broun’s inquiry from last week. The Capital Weather Gang also reported today the acting director of the NWS, Laura Furgione, disbanded the original NWS service assessment team after what an NWS spokesperson called “initial discussions with [the Department of Homeland Security] about a potential broader assessment.” This latter revelation seems to conflict with what acting director Furgione told Climate Central’s Andrew Freedman, who wrote on November 16:
"NWS Acting Director Laura Furgione told Climate Central that the formation of the service assessment had been “premature” and that she had not seen or approved a charter governing the scope of the team’s work. Furgione did not say who approved the initial decision."
So, was the NWS effort “premature” and shut down because Acting Director Furgione hadn’t approved the charter and budget or was it because of these “initial discussions” with DHS? And why would the NWS scuttle an already ongoing effort on nothing more than the “potential” for a broader review based on preliminary conversations with another department with no oversight of the NWS? Neither speaks to the notion of strong leadership of the NWS.
Needless to say, I look forward to Dr. Lubchenco’s responses to Rep. Broun’s questions, and if and when a service assessment team does come together – multi-agency or not – I hope it will be given broad license to ask the questions it deems necessary and the independence to perform its task as it sees fit. The trouble in getting it off the ground does not instill anyone with a lot of confidence, however.
The Director Speaks
In any event, there was movement from another key player in this saga yesterday. National Hurricane Center (NHC) Director Dr. Rick Knabb made his first public comments about the controversial warning issue in an interview with The Weather Channel, his former employer, who originally billed the interview as an “exclusive”. (Original tweet has been deleted, but there is a screen capture.)
[UPDATE: Thursday 11/28 10:45am: NHC Public Affairs Officer Dennis Feltgen, in an email, refuted TWC's claim of exclusivity, saying: "This was NOT an exclusive interview. Dr. Knabb provided the same interview and content to CNN, FNR, Reuters, and several other print and local TV outlets that same morning. I am reaching out to TWC right now..."
Sean Breslin, Content Coordinator for weather.com, TWC's website, apologized for the error in a tweet, saying it was an "Honest mistake".
If the interview was not an exclusive and was part of an open media day, that takes care of the questions I raised before concerning the propriety of an exclusive interview.]
Why Hurricane Warnings Weren’t Issued
Nevertheless, Dr. Knabb spoke with TWC (the entire interview is worth your time) and did answer a number of questions regarding Sandy and how the NHC and NWS handled the storm and its forecast. Asked why hurricane warnings weren’t issued for the northeast coast ahead of Sandy, Dr. Knabb laid out three “potentially unpleasant options”:
One would have been, well if it becomes post-tropical Sunday night, say a day before landfall, then if we had just gone by the books of how we do things when it becomes post-tropical — we stop advisories, take down the hurricane watches or warnings that we would have put up — that wouldn’t have been such a good thing to do, because you don’t want to change the hurricane watches and warnings mid-stream. It would have been pretty bad to put up a hurricane warning, everybody starts evacuating, and then in the middle of the event, the warnings come down. So that was one reason why we elected not to put them up in the first place, because the only thing worse would have been to put them up and then take them down.
Now, to preserve those warnings in such a scenario, when it becomes post-tropical a day or so offshore, the only other things we could have done were to continue to write advisories on it, even though it’s post-tropical. Continue to leave hurricane warnings up even though it’s post-tropical… but we would have risked breaking the dissemination system of our products and our warnings. That was a risk we weren’t too keen on taking.
Then finally the only other option we would have had was to, despite it being post-tropical for a day or so before landfall, to — for lack of a better term, fake it — call it a hurricane for a day or more before landfall.
Credibility at Stake
He goes on to say “[w]e try to call it like we see it” and argues their credibility in watches and warnings hinges on ensuring what they issue matches what they believe is going to happen. “Sandy just posed the scenario where no matter what we were going to do, it was going to have the potential for confusing people.”
I can appreciate the desire to ensure that the warnings and advisories that get posted match the weather that is expected to happen: It doesn’t make sense to issue a heavy rain advisory when you expect snow. However, a few things about that don’t sit well, and the end result was more confusion on the part of the public than was necessary.
Two Breaks From The Past
As an organization, the NHC is notorious for their keen awareness of their own limitations and the state of hurricane forecasting science, warts and all. Even a quick Google search of past NHC discussions turns up dozens of references to the average track error or the lack of confidence about an intensity forecast owing to past performance in this arena. Overall, the NHC bends over backwards to defer to historical performance and error trends, opting to err on the side of caution when those error trends suggest doing so. This quality is viewed quite favorably amongst the weather enterprise and should be a model for all who would presume to predict Mother Nature’s next move.
So, it caught me off guard that the usually very-aware-of-the-limitations NHC was so confident of its complete Sandy forecast package – track (which has improved significantly in the last three decades), intensity (which has not improved over that same time, and exact internal structure (for which no verification statistics are readily available) – as early as 72 hours out that they chose not to issue tropical storm or hurricane warnings, opting instead for the non-tropical high wind and coastal flood warnings. I would argue this was a rare public risk communication misstep on the part of the NHC, and the public in Sandy’s path would have been better served had tropical watches and warnings been used instead.
When this is brought up, there are usually two rebuttals. First, some argue that Sandy had already made the transition to a post-tropical cyclone before it made landfall; therefore, tropical storm or hurricane warnings would not have applied because they are predicated on the particular conditions coming as a result of a tropical cyclone, which by NHC’s declaration, we did not have as of a couple of hours before landfall. Second, it is argued there is no evidence to suggest tropical storm or hurricane warnings would have been any more effective than the high wind and coastal flood warnings that were posted.
Warnings Should Match the Phenomenon
On the first point of meteorological accuracy or consistency between warnings and actual events, I agree with Mike Smith who wrote four days before Sandy’s landfall that this distinction is meaningful to us meteorologists but not to most of the affected public. Readers of this blog can no doubt recall instances where straight-line winds damaged or destroyed a home, but the homeowner believes it was a tornado and usually finds a way to say so on television. If your home is flattened by a storm, it won’t matter what some meteorologist calls it – your home is still flattened.1
- NWS Warnings during Sandy
I also agree wholeheartedly with Dr. Knabb that our credibility as a science is built on consistently abiding and doing right by the best science we have available. And by our passionate nature, we want to extend a correct awareness of the workings of the atmosphere to the publics we serve. We didn’t get into the weather business to go around calling blizzards “monsoons” or sunny days “overcast”. When it is clear-cut, we should not issue warnings that do not match the phenomena at hand!
However, this extra-tropical transition business is messy, there is no scientific consensus on at exactly what point is considered to have occurred, and as the NHC regularly points out, there are limitations on how confident we can be about the track, intensity, and structure of a tropical cyclone three or four days out. Instead of deferring to the usual scientific conservatism that has served it so well in the past – likely resulting in tropical storm or hurricane warnings being posted along much of the northeast coast and prompting evacuations as early as Saturday – the NHC hung its hat on the 72-hour forecast of a post-tropical cyclone making landfall late Monday night or early Tuesday morning and relied on (and coordinated with) the local NWS weather forecast offices to post high wind and coastal flood warnings.
As a result, meteorologists spent precious time over the weekend explaining (or attempting to explain) why “Sandy wouldn’t be a hurricane but would be a ‘post-tropical cyclone’, would still be as intense, would probably be even bigger than the typical hurricane, and why people should treat these high wind and coastal flood warnings just like a hurricane warning” instead of riding folks in Sandy’s path to get off the couch, prepare, and potentially evacuate.2 Looking back, some meteorologists have taken issue with the NHC’s classification of Sandy as post-tropical at landfall. The final reanalysis may or may not back that up, but the bottom line is we were well within the margins of error for what Sandy should be called – especially when the decision was made late Friday and communicated over the weekend – and that hurricane warnings could have been used and defended after the fact.
Which is More Effective: Hurricane or High Wind/Coastal Flood Warnings?
Of course, this only makes sense if you accept that tropical storm or hurricane warnings would have garnered a better response from the public. On this point, we may have something to learn from our Canadian neighbors, who have to deal with hurricanes and post-tropical cyclones about as frequently as the northeast coast of the United States. Before 2004, the Canadian Hurricane Centre (CHC) relied exclusively on what we would call purely “impact-based” warnings – their equivalent to our high wind and coastal flood warnings – to convey the risks due to landfalling tropical and post-tropical cyclones. However, beginning with the 2004 tropical season, the CHC began issuing tropical storm and hurricane warnings, including for landfalling post-tropical cyclones. They made this shift away from impact-based warnings because the impact-based warnings were not generating the kind of response from the Canadian public they wanted, even going so far as to call the response to Hurricane Juan “lackadaisical at best”. (This may raise concern for an eventual shift here in the US toward a purely impact-based approach – a post for another time.)
Where Are We Now?
Asked in the TWC interview about how the NHC would handle a future storm like Sandy, Dr. Knabb replied:
At least in terms of having more options, not necessarily that we would do it differently next time, but at least having more options so we could potentially do it differently in whatever scenario that the atmosphere throws at us that we can’t even envision right now.
Those new options are only going to come after we learn what worked and what did not with Sandy. A month after Sandy’s landfall, we still do not have a formal NWS-centric or broader, multi-agency service assessment under way. Beyond the NWS’ performance, the entire weather enterprise – public (including emergency management), private (including broadcast meteorologists), and academic – owes itself and the publics it serves a thorough review of how we handled Sandy. In addition to the classification and warning questions facing the NHC and the broader NWS (on what was otherwise an exemplary forecast for Sandy’s track and intensity), the rest of us should be willing to assess our performance and learn from this as well. For example:
-Did the emergency managers and broadcast meteorologists take the underlying forecast information for Sandy (which was talking about record NYC surges as early as Saturday!), translate that into what their publics and viewers really needed to know (it’s going to be bad and you should leave!), and communicate it clearly, or were we too reliant on what category the NHC thought Sandy fit in and too caught up in explaining distinctions that didn’t make a difference?
-Are there gaps in our understanding about how storms make this transition, the effects of such a transition close to landfall, and how well those are handled by the various forecast models? How confident should hurricane forecasters be in model forecasts of transitioning hurricanes?
-What don’t we know about public response to warnings of various types, and how can we craft better warning messages that prompt appropriate action from a larger proportion of affected people without hyping the situation?
-And who will underwrite the efforts to answer these and other questions? The NWS service assessment process has been extended to attempt to answer some of these questions on a case-by-case basis and with an NWS-centric viewpoint; however, is an SA the best place for such work to be done or should the NWS be allowed to review its performance while independent researchers attempt to answer the broader social science questions and receive sufficient funding to do so for the good of the entire weather enterprise?
As a weather enterprise, we have a lot of work to do. The clock is ticking, and we are not doing anyone any favors by remaining on the sidelines.
UPDATED 4:53pm ET to include link to Capital Weather Gang report on discussions with DHS and to raise questions about the apparent conflict in explanations regarding the original assessment team.
1 There are important differences between this severe convective weather example (where damages due to straight-line winds and tornadoes are treated equally by insurance companies) and a landfalling tropical or post-tropical cyclone (where “hurricane deductibles” allow insurance companies to make smaller payouts if damage is due to a hurricane instead of a non-tropical or post-tropical system). If the argument is about meteorological correctness, the downstream concerns of insurance deductibles should not factor in the decision.
2 I believe this points to an inherent weakness and lack of flexibility in the NWS product suite and verification scheme as much, if not more, than problems at the NHC. Had, as Dr. Knabb mentioned, the NHC just “faked it” with Sandy, kept calling it a hurricane, and run with the hurricane warnings along the northeast coast, there would undoubtedly be calls, similar to Sen. Schumer’s, to classify (or re-classify) Sandy as something other than a hurricane. I believe the meteorologists at the NHC want to do right by the meteorology but are limited by constraints of the NWS product suite and how their forecasts and warnings will be verified. A more robust (but not necessarily larger!) suite would not have forced the them into the untenable position of having to change the wholesale approach to warning for Sandy three days out when the preponderance of the evidence suggests we as an enterprise are not that good yet. I’m told work is underway to make the suite more flexible for situations like this; I hope this work is supported at all levels of the NWS, including appropriate funding.