Table of Contents

1. OVERVIEW
2. THE PROBLEM
3. THE SOLUTION
4. CHALLENGES IN NATIONAL DATASETS MANAGEMENT
5. IMPROVEMENTS
6. OUTLOOK
7. REFERENCES

1. OVERVIEW

For several years, the National Weather Service (NWS) has been engaged in modernizing its field operations and support. A key element of this modernization is the Advanced Weather Interactive Processing System (AWIPS), which provides communications, data processing and display, and product generation functions for NWS field operations. Field deployment of AWIPS commenced in 1997 and continued through 1999. Commissioning of AWIPS, that is, certifying that it fully supports NWS operations, began early in 2000, and was completed in August 2000 (Facundo 2000).

AWIPS configuration files support a variety of functions, such as converting AFOS/WMO/AWIPS headers, defining the AWIPS menus at individual sites, directing operations of the decoders, providing map backgrounds, and organizing the storage of weather products and data.

During development and early tests, with only a few participating field sites, each site could be customized at some leisure. To support general deployment, an extensive body of so-called localization software was developed. Localization includes customizing some of the configuration files to produce menus, map backgrounds, and storage directives appropriate to each site.

In order to provide continuity in site backup, software troubleshooting, and AWIPS software enhancements and upgrades, it is appropriate that many of the configuration files be centrally maintained. Thirty-four AWIPS software configuration files and 14 sets of map background files have been designated as national datasets (Thigpen 1999, 2000).

This paper describes the management and maintenance of these AWIPS national datasets, which must function in the ever-changing environment of enhancements and upgrades to AWIPS software and hardware.

2. THE PROBLEM

Initial AWIPS deployment software suites included all appropriate configuration files. (Hereafter, we refer to these as NDM files, for National Datasets Maintenance.) This was of course necessary to support the initial localization - for data ingest, processing, and display at each field site. Difficulties first arose when upgrades began to be delivered to the field. Field sites had been encouraged to review the files and provide feedback on needed changes, as well as to make those changes locally. Unfortunately, with each release of software, locally adapted versions of some datasets were overwritten, without sites being notified that this would happen, or how to prevent it. The site's local changes were often lost, unless they had saved their files at a safe location. Even then, the newly delivered datasets had to be edited to merge and reconcile the changes, causing extra hours of downtime and many hours of troubleshooting.

The testing cycle for software releases spans many months. Even though site-requested changes were put into the NDM files maintained by developers, the versions delivered with a new software load were often considerably out of date. Field users and developers alike were frustrated by the seeming inability to get needed changes included in AWIPS, with users frequently believing that their requests were being ignored.

3. THE SOLUTION

The NDM files are summarized at http://www.awips.noaa.gov/ndm/. This site includes a copy of each current file, information on where the file resides on the field AWIPS computers, a description of the file's function and how to localize it (if necessary), and a history of modifications to the file, so users can decide if they need to download a new version. Sample portions of this Web page and a subsidiary page are shown in Figure 1.

AWIPS NATIONAL DATASETS This table provides a quick view of the AWIPS National Datasets, descriptions and the process of updating. Updates to data will be available on this page and also on NOAA1 pub/ndm shortly afterward. If sites have not sent in changes since November 1999, their changes may not be included in these updated datasets. Dataset / Date / Size Node/Location to Install How to Update File Description / Metadata File History metarStationInfo.txt 9/11/00 as1 (as2 for failover) and workstations /awips/fxa/data This file contains station information for METAR and SAO sites. It is read by the METAR decoder, which reports in its log if it doesn't find a station (but stores the decoded report, anyway). This is also used by localization (-station option) to complete MTR.spi (station plot info), for those stations which do not appear in MTR.goodness, allowing them to be plotted. Update history

Maintenance history for metarStationInfo.txt Date Additions Modifications Deletions Remarks 11 Sep 00 CXQA Qavvik Lake, NT KARL Arlington, WY KBFU Belle Fourche, WY KBIT Bitter Creek, WY KBRX Bordeaux, WY KBVR Beaver Rim, WY KCHJ Chief Joseph, WY KCMS Cemetery Separation, WY KCTD Continental Divide, WY KDHS Dead Horse, WY KDRC Deer Creek, WY KFIR First Divide, WY KGUN Gunbarrel, WY KHKS Jackson/Hawkins Fld, MS KIDV I-25 Divide, WY KIKA Inyan Kara, WY KPAT Pathfinder Hill, WY KPIN Piney Creek, WY KPUM Pumpkin Vine, WY KRIM Rim, WY KSGE Sage, WY KSHC Shute Creek, WY KSIB Sibley Peak, WY KSKL Skyline, WY KSNH Savannah/Hardin Co, TN KTMH Twenty Mile Hill, WY KVDW Vedauwoo, WY KWTR Whitaker, WY     WY metar additions 1 Sep 00     K0Y7   ... ... ... ... ... 25 Mar 99       last PCMS baseline Last update 11 Sep 00 by Fran Curnow

Figure 1. Excerpts from (top) the NDM Web page; (bottom) a history log.

Updating is complicated by the fact that this documentation and the dataset repository are maintained on an Internet site, while the datasets for field use must be accessible on the AWIPS Wide Area Network (WAN), which is isolated behind firewalls. Accordingly, downloadable files also are maintained on a dedicated server (hereafter referred to as the NDM server) within the AWIPS WAN.

Sites determine when to download updates to the national datasets based on what the update will provide in products or functionality and their unique on-site priorities. (In some cases, a download is mandated by NWS operations policy.) The sites are provided information about updates through specific notification to regional focal points and sites as appropriate. To provide users quick access to recent and pending changes, a Web page, http://www.awips.noaa.gov/ndm/natdatalog.html, was created for logging these changes (Figure 2). This log also reveals where the update resides and whether notification has been sent; the changes are numbered so that discrepancy reports can reference a specific dataset change.

Log of recent and pending updates to the AWIPS National Datasets Notices to AWIPS Regional Focal Points are pending for some of these dataset changes Date Dataset Change Remarks; NDM# Status Available 9/12/00 metarStationInfo.txt MTR.goodness textCCChelp.txt add WY metars NDM#69 notice pending NDM NOAA1 pub/ndm 9/6/00 collective_table.dat add FTHW31 CCCTAFXXX NDM#68 notice to PR, WR NDM NOAA1 pub/ndm

Figure 2. The NDM Update Log (excerpt).

Sites are encouraged to "diff" the master datasets with their version of the datasets to ensure that the updates contain the edits required locally, and to send any differences to the national datasets manager.

4. CHALLENGES IN NATIONAL DATASETS MANAGEMENT

With approximately one year of experience now behind us, we have identified a number of areas that are particularly challenging.

• Frequent internal and external changes in products, external data acquisitions, and communications challenge national management and maintenance of the national datasets and will continue to do so. Providing regional focal points and sites with specific dataset change information and concise instructions for updating, such as in the status log previously mentioned, is essential. Keeping sites informed about upcoming changes and impacts is most constructive.
  For broad dissemination to AWIPS users, notices about NDM updates and policies are sent to the awipsinfo listserver, now serving over 700 AWIPS-interested members. Problems and "how-to" questions frequently are posed, with answers coming from other sites, developers, and headquarters.
  There is no loss to field flexibility with the advent of central control, with one exception. When AWIPS software upgrades dictate a change to the format or contents of an NDM file, the site must download and install the new version at the time of software installation. It is recommended that the site download the dataset ahead of time and make any necessary local changes, after the site has submitted the changes to the national datasets manager.
• Decisions about which dataset(s) to change and what changes to make are not always clearly defined, as new products and sources arise.
  The descriptions of the national datasets provide clues for updating, and include the dependencies, i.e., how a change to one dataset affects others. For example, the metarStationInfo.txt file provides block number, ID, latitude, longitude, elevation, and location for each METAR site. One must also add the METAR ID and location to textCCChelp.txt, and the ID to afosMasterPIL.txt in order to see it in the text browser.
• Each dataset requires its own set of update instructions. These must be conveyed to the sites with each update notification, and are being added to the main NDM Web page. As noted, instructions for some datasets include localization requirements. If the site's local changes have been included in the national dataset(s), localization provides the intended results. Instructions include how to retrieve the datasets, where they go (which directories on servers and/or workstations), and which processes need to be restarted, if any, to effect the changes. Thus, another factor in managing the national dataset changes is consideration of the impact to the sites in time and complexity.
• At least two versions of AWIPS software are in field use at any given time. Since some of the NDM files are version-specific, it is necessary to manage more than one version of the national datasets at a time. Directories separate these different versions in both the master datasets repository and on the NDM server inside the AWIPS firewall. Software installation procedures provide the specific directory name for the datasets, based on the version desired.
• Timing of updates can be critical because some of the national datasets control what products sites store and send, and with what headers. Thus, sites must be notified of the need to update their datasets and how and when to make the changes. When new products are scheduled, the applicable newly updated national dataset is placed on the NDM server for sites to download. Notices are e-mailed to the AWIPS Site Support Team, regional focal points, the awipsinfo list, NWS National Centers, the AWIPS Network Control Facility, and the WSR-88D Operational Support Facility, with instructions for updating the particular dataset(s) at sites. Update procedures must be spelled out since the steps vary according to the dataset.

5. IMPROVEMENTS

Several improvements to the NDM process are in progress or identified. We wish to:

• Provide installation instructions on the NDM server for access by AWIPS field users.
• Develop scripts to obtain needed dataset updates, copy updated datasets to AWIPS servers and workstations as appropriate, and provide for localizations as needed.
• Include the AWIPS User's Manual and System Manager's Manual (SMM) in the NDM server repository, and provide scripts to download and install these on field systems. Present practice is to deliver the previous build's User's Manual on the installation CDs, along with an up-to-date printed version. The SMM is not available in soft copy on AWIPS disks at present.
• Clean up many of the text database support files, based on field input. Publicize what sites can do to help in the streamlining and cleanup of the datasets, and how the software and datasets are affected.
• Allow the use of HTTP over the AWIPS WAN to simplify downloads for field sites. For example, if the data and documentation now on the NDM Web page were available on the WAN, we would be able to provide clickable download mechanisms to field users.

6. OUTLOOK

Considerable effort has been invested in improving the AWIPS datasets maintenance service provided to National Weather Service field offices. The infrastructure and basic capabilities needed to centrally support and provide configuration files have been developed, and improvements to the process are ongoing.

As AWIPS matures and evolves, the list of files included in the national datasets will likely expand. We will need to continue to augment and improve both documentation and installation support to keep pace with these changes.

7. REFERENCES

Facundo, J., 2000: Update on Commissioning the Advanced Weather Interactive Processing System (AWIPS). Sixteenth International Conference on Interactive Information and Processing Systems for Meteorology, Oceanography, and Hydrology, Long Beach, Amer. Meteor. Soc., 291-294.

Thigpen, R., 1999: Maintenance of AWIPS Data Sets. 3 Nov 99 e-mail to awipsinfo mailing list, available at http://www.awips.noaa.gov/ndm/nov3_1999.txt.

Thigpen, R., 2000: AWIPS National Datasets Maintenance. 12 Jan 00 e-mail to awipsinfo mailing list, available at http://www.awips.noaa.gov/ndm/jan12_2000.txt.

Footnotes

(1)

Corresponding author address: Frances B. Curnow, NOAA/NWS W/OST, 1325 E/W Hwy, Silver Spring, MD 20910; e-mail: frances.curnow@noaa.gov.

This document is maintained by Joe Wakefield.

Text updated 3 Oct 00; links updated 5 Aug 03.