Implementation of the Data Acquisition Component of the WFO-Advanced 2-d Display (D2D)
G. Joanne Edwards*
NOAA Forecast Systems Laboratory
Boulder, CO
*Contract with System Technology Associates, Inc. Colorado Springs, CO 80918
Corresponding author address: G. Joanne Edwards, NOAA/ERL/FSL R/E/FS6, 325 Broadway, Boulder, CO 80303-3328
Table of Contents
The Forecast Systems Laboratory (FSL) is currently developing an AWIPS-like forecaster workstation (MacDonald et al, 1996) in support of the modernization and risk reduction activities of the National Weather Service (NWS). The workstation system, known as the WFO-Advanced is taking advantage of current hardware and software technology to integrate, manipulate and display multiple datasets from several data sources on a single workstation. The primary objective of the WFO-Advanced is to support modernized NWS Weather Forecast Office (WFO) operations. The Denver WFO will be the first to receive this prototype workstation.
One component of the WFO-Advanced workstation is the interactive 2-dimensional visualization workspace, (D2D) formerly FX-ALPHA (Bullock et al., 1994) for data access and manipulation. This paper addresses the data acquisition part of the D2D workspace. It discusses the major components of the subsystem, and the data sets that are provided for display and manipulation on the workstation.
Several methodologies were taken into consideration in designing the data acquisition subsystem: client/server methods, interprocess communications, and object oriented design.
Client/server methodology is used between a process that has a service to provide and processes requiring that service. The data acquisition subsystem uses its own flavor of this technique, in that client processes request a particular type of data from a server process, and whenever that data become available, the server automatically sends either the data or a notification to the client.
Communication between processes is handled using interprocess communications (IPC) software developed at FSL. This software is layered on top of Digital Equipment Corporations's DECmessageQ or DMQ.
Most of the data acquisition subsystem has been designed using the traditional procedural methods of flow diagrams showing both the data flow and the functional components. The interprocess communications, on the other hand, was designed using an object-oriented approach which defines a problem in terms of objects, having both data and functionality. In the WFO-Advanced, we implemented objects using C++ classes.
The primary objective of the data acquisition subsystem is to provide data from both national and local data sources to the workstation in a timely manner, and in a consistent format. To accomplish this, data acquisition was designed to be modular, in which the main functionality could be separate into processes communicating with each other via the IPC mechanism. The components of the subsystem are: Communications Servers, Communications Router, Data Controllers, Data Decoders, Outbound Request Server, and Interprocess Communications. Figure 1 shows the data acquisition subsystem components and the data flow between the functional components.
Figure 1. Data Acquisition Components
The communications servers provide an interface between the outside world and WFO-Advanced. The earlier versions of the WFO-Advanced, known as FX-ALPHA, provided this functionality by two types of servers, Local Data Management (LDM) bridge processes and an asynchronous communications server. The latest version of the WFO-Advanced D2D ingest also includes one or more synchronous communications servers.
The LDM bridges provided both national and local datasets to WFO-Advanced from FSL's NIMBUS (Pedigo et al, 1994) using the LDM system developed by Unidata. The workstation system that is deployed to Denver will receive only experimental and locally generated products in this manner. LDM is a collection of cooperating programs designed to enable the capture, processing and distribution of data. The LDM bridges act as clients, requesting data or notifications from an LDM server running on WFO-Advanced. When data become available from NIMBUS, the LDM server sends either the data or a notification of data arrival to the appropriate LDM bridge. The bridge then passes on the information to one or more decoders via router and controller processes.
Asynchronous communications servers provide the interface between external, event driven observation platforms and the WFO-Advanced. The servers configure the port, "listen" for data on the line, read in the data when they become available, and pass on the data to decoders. There is no handshaking between the servers and the external system.
WFO-Advanced was designed to run at the WFOs as a stand-alone system, ingesting data via communications servers developed specifically for each data source. New synchronous data feeds, such as the Denver WSR-88D and the Satellite Broadcast Network (SBN) will be serviced by one or more synchronous communications servers (Sync Comms Servers). These communications servers will interface with the communications devices, ingest the data, and pass on information about the data to a communications router. These servers will also detect line or device anomalies and relay that information to the workstation display area.
The communications router (Comms Router) receives data passively from the communications servers. There is no client/server relationship here in that the router does not request the data. On receipt of data, the router determines which of the data controllers are to receive the data using pattern-matching techniques. The information is then passed on to the appropriate controllers using the IPC classes.
The data controllers route data from the communications router to the appropriate decoder processes. The controllers are classified as both client and server. They accept requests for data from the decoders and pass on the requests to the communications router. This causes the router to continuously send data to the controllers. Thus, the controllers are clients of the router and servers for the decoders.
The data decoders are designed to be clients of the data controllers. The decoders send requests for data to the appropriate controllers, and receive data or notifications back from the controllers. There are two types of decoders. One type receives data via the IPC, decodes the data and stores the new product in a consistent format for access by the workstation. The other type of decoder receives notifications of data arrival, and based on information about the data, retrieves data from data directories, decodes the data and stores the product in a new directory. Currently, products are stored in three formats, NetCDF, binary and text files. We found that workstation functions could access much of the data stored in binary files faster than the same data stored in NetCDF. We reserved the NetCDF files mainly for access by analysis and forecast models. Text products are not stored by the ingest decoders but are sent to a database machine where the products are stored in the database.
The outbound request server accepts requests from the workstation, or on behalf of the workstation, and passes on the request to the appropriate front-end communications server. The communications server then sends the request to the external source via the communications link. The outbound request server also keeps track of the requests and informs the workstation in the event the request(s) cannot be honored. The first application is the WSR-88D data ingest at the Denver WFO.
The IPC is implemented as a library of C++ classes layered on the DMQ communications mechanism. All processes within the WFO-Advanced communicate with each other using the IPC class library. When one process wishes to send a message to another, the message is sent to the receiver's queue address, which is established by DMQ. Receiving processes use the IPC classes to issue a wait on their queue, and receive any data forwarded to that queue.
The Denver WFO-Advanced will receive both national and local data from several data sources in various formats: satellite imagery, text and redbook graphics from the Satellite Broadcast Network (SBN); radar data from the WSR-88D; mesonet data from the Urban Drainage and Flood Control District (UDFCD) and the Colorado Department of Transportation (CDOT); locally generated products and experimental products from FSL via NIMBUS.
The D2D workspace in the WFO-Advanced will receive a number of datasets via the Satellite Broadcast Network (SBN). These datasets include the AWIPS data stream consisting of Surface Aviation Observations (SAOs), Automated Surface Observing Systems (ASOS), radiosonde, profiler, text and gridded data from the National Weather Service Telecommunications Gateway (NWSTG); and NESDIS satellite imagery.
The SBN is a Ku-band satellite network covering the Conterminous United States (CONUS), Alaska and Hawaii. Data are collected at a central facility, formatted into a transmission format, and transmitted to the satellite uplink. A satellite downlink receiver at the Denver WFO ingests the satellite broadcast. A synchronous communications server running on the WFO-Advanced will read the data, and pass on information about the data to the communications router.
The WFO-Advanced will acquire WSR-88D products from the Radar Product Generator (RPG) via the SBE X.25 communications interface. The SBE software operates with the SBE series of multichannel, high-speed synchronous communications controllers, enabling communications with packet-switching networks. The SBE controller handles most of the communications processing such as the handshaking, and assembly of packets, thus freeing up the host machine to do other processing. A synchronous communications server running on D2D will interface with the Network Layer Interface (NLI) thus enabling the communications server to access the packet layer of the X.25 protocol.
The WSR-88D data, a bit-synchronous stream of data are received from the RPG at a rate of up to 56kbps. On start-up, the radar synchronous communications server (Radar Sync Comms) sends out a routine product set (RPS) list containing the products that are to be routinely sent by the RPG. Once the data are received on WFO-Advanced, the server will store the data, formulate a notification, and send the notification to the communications router.
In addition to data receipt, the ingest component of D2D also provides the capability for forecasters to request radar data on either a short-term basis, or more routinely, depending on weather conditions. D2D ingest will receive the request from the workstation and send out the request to the RPG via the outbound request server. D2D ingest will also send out a new RPS list when it detects a change in weather mode based on status information received from the RPG.
The Urban Drainage and Flood Control District (UDFCD) provides Automated Local Evaluation in Real Time (ALERT) data to the WFO-Advanced on a continuous basis. The ALERT system consists of a network of sensors, gauges, and radio repeaters. In Denver, this system is monitored and maintained by the UDFCD. The sensors include precipitation gauges, river and reservoir stage (water level), and weather sensors. The weather sensors measure such phenomena as wind speed and direction, air temperature, relative humidity, barometric pressure, and solar radiation. As events occur, the sensors transmit the data to the base station at the Denver UDFCD, where they are ingested, decoded, and the resultant products are made available to users.
An asynchronous communications server (Async Comms Server), running on the WFO-Advanced, receives the data from the UDFCD over a dedicated telephone line at 9600 bps, and passes on the data to the communications router.
The Colorado Department of Transportation (CDOT) system consists of a network of Maintenance Sections or base stations, Remote Sensing Units (RSUs), and sensors owned and operated by CDOT and Surface Systems, Inc. (SSI).
There are eight Maintenance Sections within Colorado that act as repositories for data from 38 RSUs. The Maintenance Section in Denver receives data from 14 RSUs. Each RSU ingests data from five sensors -one surface and four subsurface - that measure such parameters as air temperature, precipitation, dewpoint temperature, and wind speed and direction.
Each Maintenance Section dials out to each of its RSUs and ingests the data. The CDOT interfaces with each Maintenance Section over a dedicated, leased line. A process running at the CDOT polls each Maintenance Section CPU every 15 minutes and ingests the data.
The WFO-Advanced communicates with the CDOT system over a 14.4 kbps dedicated line. At 15-minute intervals, CDOT transmits the data for each Maintenance Section to the WFO-Advanced where it is stored on disk for later decoding.
Analyses and forecasts from the Local Analysis and Prediction System (LAPS) are generated on WFO-Advanced (Schultz, 1996). A LAPS ingest process reads SAOs, mesonet, satellite, radar and profiler data from the WFO-Advanced ingest storage, and stores the data for access by the LAPS analysis.
LAPS reads the data from ingest format and performs an analysis. The LAPS analysis cycle is currently once per hour, but will be increased to once every half hour. Once the analyses are stored, a transfer process extracts pertinent fields from these files, and creates a single file for access by the workstation.
A LAPS forecast process that runs once per day at 0600 UTC reads the analyses and generates forecasts out to 18 hours. Once these forecasts are generated, a transfer process extracts pertinent fields from all of the forecasts and generates a single file for access by the workstation.
The WFO-Advanced will receive experimental data from FSL's NIMBUS. Current plans are to have the data transmitted from FSL to Denver's WFO-Advanced workstation over the T1 link using the LDM system. The experimental datasets consist of mesonet data and FSL generated model output.
The objective of the D2D workspace of the WFO-Advanced is to demonstrate the capability to access, integrate, manipulate and display data from multiple data sources in different formats on a single workstation. The data acquisition component of D2D, which was designed using client/server relationships and interprocess communications, provides the various datasets in a timely manner and in a consistent format for quick and easy access by workstation functions in real-time.
Bullock, C. S. and U. H. Grote, 1994: FX-ALPHA: A new FSL Workstation. Preprints, Tenth International Conference on Interactive Information and Processing Systems for Meteorology, Oceanography, and Hydrology, Nashville, TN. American Meteorological Society, Boston, MA, pp. 354-357.
Lipschutz, R. G. J. Edwards, D. L. Davis, 1991. FSL's NEXRAD Product Interface Subsystem. Preprints, Seventh International Conference on Interactive Information and Processing Systems for Meteorology, Hydrology and Oceanography, New Orleans, LA. American Meteorological Society, Boston, MA, pp. 167-172.
MacDonald, A. E., and J. S. Wakefield, 1996: WFO-Advanced: An AWIPS-like Prototype Forecaster Workstation. Preprints, Twelfth International Conference on Interactive Information and Processing Systems for Meteorology, Oceanography and Hydrology,Atlanta, GA. American Meteorological Society, Boston, MA, in this preprint volume.
Pedigo, Jeffrey S., 1994. Real-Time Acquisition and Management of Meteorological Data - New Systems Being Developed at the Forecast Systems Laboratory. Preprints, Tenth International Conference on Interactive Information Processing Systems, Nashville, TN. American Meteorological Society, Boston, MA. pp. 330-335.
Schultz, Paul, 1996: Local Data Analysis and the Mesoscale Model on the WFO-Advanced Workstation. Preprints, Twelfth International Conference on Interactive Information and Processing Systems for Meteorology, Oceanography, and Hydrology, Atlanta, GA, American Meteorological Society, in this preprint volume.
Last updated 17 Nov 95.