In May 1996 FSL began deployment of WFO-Advanced to the Denver NWS Forecast Office (WSFO) for operational testing. Two workstations were initially deployed in order to identify problems associated with operating at a location outside FSL using operational data feeds for satellite, numerical models, text, and the WSR-88D radar. Later, two more workstations were installed so that all of the staff in operational positions at the WSFO could use WFO-Advanced workstations. It is expected that WFO-Advanced will fully replace the current operational system (DARE) in the near future.
FSL is evaluating aspects of the deployment to Denver and forecasters' impressions of WFO-Advanced during installation and initial operational use. These evaluations will provide the NWS with information relevant to future deployment of AWIPS operational systems.
The D2D introduction lasted about 2 hours and included a broad overview of D2D given by the trainer to one or two trainees. A brief discussion of the menu contents and display functionality was given. During the self-paced training, trainees worked through the D2D User's Guide, containing information and illustrations of the D2D interface as well as practice modules on the various D2D components. The trainer was available for questions or problems during this phase of the training, which typically took 6 to 8 hours to complete. An outline of key points was given to the trainees as a self check and to ensure they were prepared for the one-on-one session. The third phase of the D2D training lasting about 4 hours, was a one-on-one (or one-on-two) session with the trainer reviewing key points about D2D. The trainer followed the outline of these key points during this phase of the training. Discussion of features and capabilities was encouraged and typically generated related questions or problems that the trainee experienced during the self-paced training.
Forecasters found all three phases of training to be useful, and they agreed that the presentation level of the training material was appropriate (i.e., neither too easy nor too difficult). Forecasters indicated being comfortable using the workstation directly after training as well as at the end of the initial evaluation period (approximately three months later). However, they also indicated a higher level of comfort using the graphics/image displays than using the text display. This later finding is likely due, in part, to the fact that several components of the text display were not available during the initial training period, and to the fact that there have been many changes made to the text display subsequent to the initial training. Additionally, the WFO-Advanced Display 2-Dimensional (D2D) User's Guide was very well received by the forecasters.
As part of the initial training, forecasters were asked, immediately after completing the third phase of training, to work through a set of 10 exercises. These exercises represented a sampling of both basic and advanced workstation functions. Forecasters' performance on the exercises showed that they generally did well executing basic workstation functions (e.g., loading, overlaying, or toggling products, zooming, etc.), but often needed help in using some of the more advanced functions (e.g., setting up a point sounding or a cross section in the volume browser, editing a color table, etc.).
Following the initial training, staff-wide system training on the Product and Data Ingest Restart Monitors took place over a 3-week period. This consisted of a one-on-one description and demonstration with each forecaster, and typically lasted 15 to 30 minutes.
The Denver forecast staff were asked to rate, on a five-point scale, aspects of the WFO-Advanced workstation's user interface (see Grote and Bullock, 1997), data availability, analysis, and display, and product generation capabilities. Most importantly, they were asked to rate these capabilities as they compared with the operational systems most NWS personnel currently use to complete their shift duties (e.g., AFOS, PUP, SWIS, PCGRIDDS, NAWIPS, etc.). The rating scale was worded in terms of the impact on the forecaster's ability to meet job requirements and ranged from a low of one (1): "Frequently impedes your ability to meet the requirements, responsibilities of your job; likely to lead to degradation of job performance," to an "adequate" mid range of three (3): "Consistently enables you to meet the requirements, responsibilities of your job; does not lead to degradation nor enhancement of job performance," to a high of five (5): "Consistently enhances your ability to meet the requirements, responsibilities of your job; likely to lead to enhanced job performance."
Rated aspects of the WFO-Advanced user interface included the five-pane display, the graphic/image display menus and dialog boxes, the Volume Browser, and the text display menus and dialog boxes. In general, the user interface was rated as frequently enhancing the forecasters' ability to meet their job requirements (representing average ratings between 3.9 and 4.6). The five-pane display was rated particularly high (4.6). Forecasters reported they use the small panes for data monitoring and quick recall of data displays. One forecaster explicitly compared the five-pane display to a multiple window setup, stating that he did not "have time for window management in slow weather situations, much less in fast paced weather." The text display menus and dialog boxes received the lowest average rating of 3.9 (which is well above the "adequate" rating).
Forecasters were asked to rate the data availability for the graphics/image and for the text display, and to rate the ability of WFO-Advanced to perform WSR-88D Principal User Processor (PUP) functions. The availability of products and data on the graphics/image displays was rated between frequently and consistently enhancing forecasters' ability to meet job requirements (an average rating of 4.8). The availability of data and products on the text display was rated somewhat lower (3.9), but fell in the category of frequently enhancing forecasters' ability to meet job requirements. Comments on data availability pointed to problems with the timeliness of the Satellite Broadcast Network (SBN) and to problems with the availability of particular text products. (Many problems with text delivery were uncovered and corrected during this shakedown period.)
Nearly all of the forecasters rated WFO-Advanced's ability to perform PUP functions as consistently enhancing their ability to meet job requirements (an average rating of 4.9 on the 5-point scale). When asked to comment about the ability of WFO-Advanced to perform PUP functions, one forecaster responded that WFO-Advanced was "at least 1000% better than the PUP," and stated that "reading the raw data is simple and leads to much better performance and better warnings in severe weather situations."
WFO-Advanced's data analysis capabilities rated by the forecasters included contouring gridded and point data, deriving parameters from gridded data, combining radar reflectivity and velocity products, combining images, and using the interactive Skew-T application. With the exception of the latter, the data analysis capabilities were very highly rated, with ratings falling between frequently and consistently enhancing forecasters' ability to meet job requirements (average ratings between 4.4 and 4.8). Forecasters especially liked the combined radar reflectivity and velocity image products, with several commenting on the power of being able to quickly and easily toggle between the two products. The forecasters also liked the capability to combine other images, especially the ability to combine satellite with radar.
The Skew-T application received the lowest rating among the data analysis capabilities, being rated as about adequate (an average rating of 2.8). There were many comments about the problems with the Skew-T application, the consensus being that the application is still immature. Several forecasters did note, however, that they liked that the application had been integrated into WFO-Advanced (as opposed to having to "shuttle over" to a PC to run the program).
Rated data display capabilities included the ability to overlay graphics, overlay graphics and images, looping, zooming, and panning, the ability to fade between combined images, and the Volume Browser's time/height section and cross-section features. All of these data display capabilities were highly regarded by most of the forecasters, with mean ratings between frequently and consistently enhancing their ability to meet their job requirements (average ratings between 4.3 and 4.9). Overlaying graphics, overlaying graphics and images, looping, and zooming all received ratings of 4.0 or higher from all forecasters. One forecaster commented that he thought zooming(2) was the "best feature of WFO Advanced." Other forecasters singled out the time/height and cross section features of the Volume Browser for high praise. About the former, one forecaster wrote that the fact that the time/height section "loads as many fields as you want makes it superior to PCGRIDDS." Another wrote that the cross section is "very fast; very detailed; easier than PCGRIDDS."
Forecasters were asked to rate the product generation capabilities of WFO-Advanced with regard to issuing routine text products and using WARNGEN to generate watches and warnings. The WFO-Advanced text-editing capabilities for issuing routine text products were rated as close to being able to frequently enhance the ability to meet job requirements (an average rating of 3.9). On the positive side, forecasters felt that the text editor was a great improvement and noted especially the spell checker, word search/replace, word-wrap, and keyboard functions as improvements. One forecaster commented that the multiple text windows and the ability to edit between them as "a key improvement over existing systems." On the negative side, forecasters noted the slow keyboard response, that is, there would be a delay between their typing and the onscreen response (this problem has since been fixed), and the need for enhancements such as procedures and special text macros. (These have since been added to the workstation.)
Forecasters rated the WARNGEN function for generating watches and warnings as frequently enhancing their ability to meet their job requirements (an average rating of 4.5). Forecasters noted that they liked the ability to generate warnings off the actual data display and liked being able to have other data overlaid with radar when issuing warnings. They also commented that WARNGEN simplifies and speeds up the warning process. One forecaster, called WARNGEN "one of the best features of the system," adding that "especially important are the integration going from radar data straight to a usable text, the need for only limited editing, and access to a detailed geographic database." Criticisms of WARNGEN included the need for WARNGEN to be more stable, enhancements including the ability to click on a zone or county to be included in a warning, and the ability to deal with multiple storms or lines of storms.
TABLE. Distribution of WFO-Advanced product loads by product category during August 1996 from the Denver WSFO.
----------------------------------------------------
Product Type Product Loads Product Swaps sum of
Mean sd Mean sd Means
Model Graph. 484 131 174 87 658
Radar 149 98 147 101 296
Satellite 72 16 37 14 109
Surface Obs 35 16 33 15 68
Vertical 29 28 6 8 35
Lightning 14 8 14 11 28
Upper Air 10 6 2 5 12
* Model Snd. 36 33 10 10 46
* Model Img. 15 8 4 4 19
Model Xsec 13 11 2 6 15
* Model Xim. 3 1 0.4 1 4
* Space Load 7 2 0.1 0.6 7
Watch/Warn 3 2 0.2 1 3
Applications 6 5 1 2 7
Other 17 11 15 12 32
Totals 893 446 1339
----------------------------------------------------
Product "loads" and product "swaps" are listed separately in the Table. Products can be initially loaded into the large viewing pane either by selections from the menus or procedures. Products can be swapped into the small panes then swapped back into the large pane. The swapping mechanism accounts for one third of the products selected for viewing in the large pane in August. In the case of radar, surface and lightning products, there is nearly a one-to-one correspondence between original product loads and subsequent swaps back to the large pane. This indicates forecasters will frequently continue to monitor some products (in the small panes) while viewing other products in the large pane. This finding supports the forecasters' positive feedback on the five-pane display.Product usage patterns by product type compared favorably to the DARE warm-season product use. Model graphics and radar were loaded more frequently when compared to the Kucera and Roberts (1995) study. The increase in radar loads may be attributed to more active convective weather this season. It is uncertain why model graphics were more frequently loaded. A thorough evaluation of product use will be conducted once DARE is fully removed from the office.
Kucera, P.C. and W.F. Roberts, 1995: Warm Season Product Usage Patterns from the DARE Workstations at the Denver and Norman WSFOs. Preprints, 14th Conf. on Weather Analysis and Forecasting, 15-20 Jan 1995, Dallas, TX, Amer. Metor. Soc., 101-107.
Lusk, C.M., W.F. Roberts, E.J. Steiner, L.E. Johnson, and D. Clark, 1992: Evaluation of Pre-AWIPS Training: Interim Report 1. NWS internal report, Boulder, CO., 74pp.
MacDonald, A.E., and J.S. Wakefield, 1996: WFO Advanced: An AWIPS-like Prototype Forecaster Workstation. Preprints, 12th Conf. on Interactive Information and Processing Systems, Atlanta, Amer. Meteor. Soc. 190-193.
Roberts, W.F., L.E. Johnson, P.C. Kucera, C.M. Lusk, D. Salisbury, D.C. Walker, 1996: An Overview of Evaluation Results from the WFO-Advanced Realtime Forecast Exercise: RT95, FSL Evaluation Team Report, 26 pp.