Figure 1. Data monitor summary page.
Fiscal year 1980 saw the birth of an organization dedicated to improving short-term weather forecasting. The Prototype Regional Observing and Forecasting System (later renamed Program for Regional Observing and Forecasting Services, both using the acronym PROFS) was designed to be a technology transfer organization, providing a bridge between research and operations.
In 1988, the Forecast Systems Laboratory (FSL) was created by merging PROFS with some other groups in the Environmental Research Laboratories. The focus of the work did not change; a key project thread throughout the 1980s and 1990s was work with the National Weather Service (NWS) building and demonstrating functional prototypes of improved field office support systems, principally the Advanced Weather Interactive Processing System (AWIPS). In the late 1980s, the Denver AWIPS Risk Reduction and Requirements Evaluation (DARE) system was installed in the Denver and Norman Weather Service Forecast Offices (WSFOs). Through the DARE experience (Bullock et al. 1988), NWS was able to refine the requirements for AWIPS capabilities, thereby significantly improving the forecaster workstation before it was ever placed in field operations.
As we approach the end of the 1990s, this nearly two decades of effort is reaching full fruition, as FSL's WFO-Advanced(2) workstation software is now being installed around the country as the data display component of AWIPS.
FSL's history of over ten years of operational risk reduction testing at the Denver WSFO has resulted in the development of an effective technology transfer process. Procedures for installation, upgrade, monitoring, and support of operational systems are being shared with NWS and the AWIPS contractor, PRC Inc., as they assume their responsibilities for operational support of the over 100 future AWIPS field sites.
When installing a new user system, two key principles must be addressed:
A complete meteorological support system includes both hardware and software. Accommodating new hardware can be difficult, as both space and power considerations arise. NWS has extensive experience in this area, and has been planning for AWIPS hardware installation for some time. FSL performed extensive tests of PRC's network wiring design (results of which led to a change in the selection of hub hardware for field installation). Other valuable experience at Denver included location of components, based on cable limits, placement of workstations, and so on.
Initial software installation can be done at a staging site, with pre-loaded disks shipped with the hardware, or can be done on-site, loading from tape. FSL has generally used the former method, and AWIPS installation also is done in this way.
One hopes that any new system will incorporate all capabilities of the one(s) being replaced. In some cases, however, components of the new system may be incomplete or local customizations of the old system may not have been included in the new software. Depending on the significance of these deficiencies, parts of the legacy system(s) may need to be retained.
When presented with something new, some people are reluctant to change, especially if the old, comfortable way of doing things meets most needs. We have found that during a transition period, it is difficult to persuade some users entirely to forsake the old system. While some spin-up period is needed in order for operations to continue while staff come up to speed with the new way of doing things, the end of the transition period should not be determined by when all users have ceased use of the old. Rather, the decision should be based on operations support considerations: users should begin to use the new system once it is operating reliably and has demonstrated the capability to support office operations. Legacy systems should be retained only as necessary to maintain operations.
All systems are subject to upgrade, of both hardware and software, either to remain current with changes in observing systems and datasets or to enhance capability.
Just as with initial installation, the most important principle in accommodating upgrades is that disruption of operations must be minimized during the upgrade process. There are at least two ways to accomplish this, both of which have been employed by FSL.
First, running redundant systems allows one to shut down part of an office for upgrade without crippling operations. During the first year and a half of running WFO-Advanced in Denver, we used two parallel data ingest subsystems and databases, with each supporting half of the WFO's workstations. This proved quite successful for upgrade (and operational redundancy) purposes, though it did cause some difficulties with synchronizing datasets and user files.
When WFO-Advanced became part of AWIPS Build 3, we adopted PRC's redundancy strategy, which features dual-ported, mirrored disks and an automatic fail-over mechanism for data ingest processes. A similar upgrade strategy is possible here, as a manual fail-over can be initiated, then new software can be installed while the backup systems are running the office.
The second approach to minimizing interruption of operations is to allow enough headroom on the system's disks to install new software alongside the old. Then, the old software can be shut down and the new started, requiring only a brief outage. Further, in many cases this can be done one station at a time, while other stations continue to run.
Regardless of the upgrade method chosen, it has proven most effective to develop scripts to support the process. It seems that after every upgrade, one or two items show up that were overlooked. By putting as much of the process as possible in scripts, such oversights are less likely. (Even with scripts, it is still important to maintain a checklist of required steps. Keeping in mind the goal of minimizing operational impacts, it pays to take care and get it right the first time.)
A second very important upgrade principle is that any modifications and customizations to the system made by the local staff must be extracted and moved into the new version. This would include both office conventions (e.g., map backgrounds, text templates, color tables) and individual user settings (scripts and procedures, preferences, etc.).
FSL has developed on-site product and process monitors (Wakefield et al. 1997), which have proven to be a tremendous help to the local office staff, allowing them to be mostly self-sufficient. These monitors provide a way for forecasters and technicians to track data ingest and restart subsystems if necessary, in most cases without having to call on outside help.
The AWIPS data/product monitor is designed for use by NWS field office staff. Using World Wide Web technology, a script runs every ten minutes on the data ingest machines to collect information on the timeliness and completeness of some 40 datasets, and then to build a set of Web pages to present this information to the viewer. A summary page (Figure 1) shows the status of data timeliness in several categories, plus disk usage. Each "State" symbol echoes the worst state of the category's components, which the user can view by clicking on the link (underlined text). In the example here, a caution symbol is seen on the Point Data link. A click there will produce a detail page with lightning, METAR, profiler, and RAOB entries, at least one of which is in "caution" state (indicating that the data are older than a specified threshold). This may be due to a problem with a decoder or a data ingest process at the local site, a failure somewhere in the data distribution system, or lack of data at the source.
Figure 1. Data monitor summary page.
Most problems in the local ingest suite can be detected by the process monitor, shown in Figure 2, which keeps track of the various ingest processes to see if they are up and running. (The current version cannot detect run-away processes, or those that are up, but not receiving any messages.) Since there are over two dozen ingest processes, distributed among three processors, a summary mechanism is used, just as for the product monitor. Note also that a similar appearance is used, with both color and icon indicating the process state.
Figure 2. Ingest process monitor.
If a process is down, the user can click on the "down" icon or on a restart button (not shown), to show a menu of data ingest restart choices. Users can also review an on-line log from this monitor.
Together, these monitors and the restart mechanism allow NWS field office staff to identify and recover from nearly all problems with the data ingest system. Intractable problems can be referred to a central help site, the AWIPS Network Control Facility (NCF - Thigpen 1997; Moore et al. 1998).
Information from the monitors is automatically forwarded to the NCF, where operators are able to perform more extensive trouble-shooting of the ingest and data storage systems.
FSL provides operations support at Denver in two ways. Since the installation of the Denver WFO-Advanced system in May 1996, FSL has maintained on-call support. A systems administrator has been on call for hardware problems, and a member of the development staff has been available for help with data and display system problems. As the software and monitors have improved, and as the Denver staff have become more familiar with the system, the number of calls has dropped to near zero. The hardware has been exceptionally reliable; as this paper is written (September 1997), parts of the Denver system have been up continually since our last software upgrade a period of over four months; the overall hardware down-time rate is less than 0.01% (less than five minutes per month).
When the system was installed, Denver forecasters and technicians were trained on its use (Roberts et al. 1997). A detailed Users Guide (FSL 1997) is available to users, in both printed and electronic form, for reference or refresher self training. The AWIPS User's Guide is based heavily on FSL's version.
FSL has also contributed a great deal of information to NWS and PRC which has been incorporated into the AWIPS System Manager's Manual. Experiences of on-call staff were collected in a troubleshooting guide. Although the AWIPS configuration differs from what was at Denver during most of the time when this information was being gathered, a considerable amount of it was readily adapted for AWIPS system support.
Final system acceptance testing of AWIPS Build 3.0 was completed in September 1997. Seven sites (Salt Lake City, Pittsburgh, Topeka, Dodge City, and Goodland WFOs, and Colorado Basin and Missouri Basin River Forecast Centers) were to receive this software in October 1997 for an Operational Test and Evaluation through November. Build 3.1 should be installed at an additional dozen sites by the time the paper is presented at the Phoenix conference in January. Twenty more sites will be receiving this software in early 1998.
AWIPS Build 4 software was scheduled to be delivered from FSL to NWS on 15 December 1997, and should be deployed to the field beginning in May 1998 (both as an upgrade at the Build 3 sites and at the rest of the AWIPS sites nationwide).
The experience gained during ten years of operational risk reduction activities at FSL has paid off in helping this major transition in NWS operations be as smooth as possible. As evidenced at a Build 3 User's Forum in August 1997 (C. Bullock 1997, personal communication), field offices now look forward to the arrival of the new system, rather than being concerned over disruption of their routine procedures.
FSL, 1997: WFO-Advanced Display 2-Dimensional User's Guide. NOAA/ERL/FSL, Boulder, Colorado, April 1997, 235 pp. On-line version available at http://www.fsl.noaa.gov/~osborn/D2DUG_ToCFigs.html.
Moore, E., M. Hicklin, C. Williamson, and L. Johnson, 1998: Advanced Weather Interactive Processing System: Operations Support in the AWIPS Era. Fourteenth International Conference on Interactive Information and Processing Systems for Meteorology, Oceanography, and Hydrology, Phoenix, Amer. Meteor. Soc., in this volume.
Roberts, W. F., P. C. Kucera, C. M. Lusk, and D. C. Walker, 1997: Operational Use of WFO-Advanced at the Denver WSFO. Thirteenth International Conference on Interactive Information and Processing Systems for Meteorology, Oceanography, and Hydrology, Long Beach, Amer. Meteor. Soc., 340-343.
Thigpen, R. K., 1997: The National Weather Service AWIPS Network Control Facility. Thirteenth International Conference on Interactive Information and Processing Systems for Meteorology, Oceanography, and Hydrology, Long Beach, Amer. Meteor. Soc., 128-130.
Wakefield, J. S., R. J. Kahn, and B. L. Moore, 1997: Monitoring an Operational Weather Forecast System Using the World Wide Web. Thirteenth International Conference on Interactive Information and Processing Systems for Meteorology, Oceanography, and Hydrology, Long Beach, Amer. Meteor. Soc., 336-339.