Table of Contents 
Previous Chapter
CHAPTER 6 Data Acquisition
The WFO-Advanced team has developed software to acquire data to support the forecasting functions of the WFO. Software has been developed to acquire data from or via the WSR-88D radar, the NOAAPORT data feed, and various local data sources unique to the WFO at the Denver site.
NOAAPORT, also known as the AWIPS Satellite Broadcast Network (SBN), provides a central data stream through a PRC-developed firmware product known as a Communications Processor (CP). The CP acquires data from the demodulated signal off a satellite broadcast feed, and concatenates and distributes data to several data server machines. WFO-Advanced sees the CP software as a "black box;" software development has been directed towards creating a process to acquire data from the CP (simulating a DWB-like connection) and assimilating these data into the WFO-Advanced data acquisition scheme (see Section 7.1.1, "System Design").
The WSR-88D interface is an extension of work performed over the last several years at FSL. FSL was able to develop certified Class I interfaces for radar ingest and display on the DARE II operational system at the Denver WSFO and the Pre-AWIPS system installed at the Norman, OK WSFO. Open-systems-compliant hardware was used for the WSR-88D interface. The ongoing efforts include a multiple-radar interface as well as a dial-out capability to acquire data from non-associated radars.
The third effort of the data acquisition system includes the ability to acquire data from local data sources. Although the Local Data Acquisition and Dissemination (LDAD) function is still being defined for AWIPS, WFO-Advanced includes a subsystem which is complemented by an effort within FSL which performs LDAD functions. This effort includes the use of PCs (outside of the AWIPS firewall) to acquire data. FTP is used to move the data between the PCs and the WFO-Advanced system. These data are currently being decoded for use by the graphics display. Efforts are underway to define a generic database for storage and retrieval of these data.
Future data acquisition and dissemination projects include the acquisition of ASOS data from a dedicated link, as well as asynchronous acquisition of ASOS data. NOAA Weather Wire dissemination will also be added to WFO-Advanced, as will the distribution of data over the AWIPS Communication Network (ACN).
The WFO-Advanced acquisition server process (acqserver) is the interface to the NOAAPORT Receive System (NRS) Communications Processor (CP). The NRS reads data streams from the AWIPS Satellite Broadcast Network (SBN) and sends them to a configured host data server over an ethernet based Local Area Network (LAN). The acqserver process runs on the data server host machine. The two data streams being sent from the CP are GOES_WEST (either EAST or WEST can be configured) imagery data and NWSTG (NWS Telecommunications Gateway, including text, graphics, observations, and model grids). These data streams are routed through the AWIPS Network Control Facility (NCF).
The GOES channel uses a T1 connection which has a data rate of 1.536 Mbps while the NWSTG data channel is 128 kbps. The NRS uses a HP 9000/747 computer running the HP-RT real-time operating system. HP-RT provides a UNIX-compatible environment including the UNIX shells, most commands, and IPC facilities (shared memory, semaphores, and message queues), as well as networking services such as FTP and NFS. The NRS software performs receive-only data acquisition, packet reassembly, product error control, SBN monitoring, and product transfer to configured host data servers. Its functionality and configuration is covered in the NRS User's Manual provided by PRC.
The acqserver process acquires weather data from the CP across a direct socket connection to the acq_client process of the NRS. This ties into the NRS system's DWB transactions to set up the connection versus the standard configuration of using either NFS or FTP transfers (as documented in the NRS User's Manual). The acqserver process was adapted from a sample server implementation sent to us from PRC. The main added functionality was product identification and storage into our hierarchical directory structure. A product pattern-matching design was implemented. A configuration file is used to specify where a product matching a regular expression pattern needs to be stored in the data management directory hierarchy. Products are sent from the CP in 4000 byte packets which contain a header identifying the product type and category. The current types being used are NWSTG and GOES_WEST. The product categories are:
- TEXT
- GRAPHIC
- GRID
- POINT
- OTHER (text)
- IMAGE
The acqserver process reads the patterns configuration file (acq_patterns.txt) to determine where products that match a pattern are stored. The directories are relative to the start of the data acquisition directory structure ($FXA_DATA). A product matching more than one pattern may be stored in multiple directories. Hard links to the different directories are used if possible (when they are within the same file system) if multiple patterns match. Using hard links saves on disk I/O and disk space. Here are some sample lines from the configuration file:
GRAPHIC ^[PQ].* /ispan/graph
IMAGE ^T.* /goes
GRID ^[YZ].[A-MO-SU-Z].* /Grid/SBN/Raw
POINT .*IUPT(0[1-4]).*|.*IUPT40.* /ispan/bufr
POINT ^IUST[0-9][0-9].* /ispan/bufr/raob
TEXT ^SFUS41.* /ispan/binLightning
The 4000-byte packets sent by the CP are written (appended) to a time-stamped file as they arrive. The file name also contains the WMO header identification string. The acqserver process assumes the data are being sent in proper sequential order; no validation of the header sequence numbers is performed. After the last packet of a particular product arrives, the file is closed and a notification is sent to the Comms Router process.
The patterns configuration file can be changed at run time and will be re-read by the acqserver process on receipt of the HUP signal.
Products are sent from the CP to the host running the acqserver processes. The data are sent in 4000-byte packets plus a header. The header contains an indication of the first and last packet of every product sent (as well as sequence numbers for every intermediate packet.) The primary function added to the original PRC sample server implementation was storeProduct() which performs the pattern matching and appropriate product storage. After complete products are stored, a notification is sent to the CommsRouter process which through the DataControllers delivers the notification to the Decoders. The storeProduct function is called for each received packet and is passed the header, the buffer containing all or part of a product, and the time it was received. The first 11 characters of GOES products are used to perform pattern matching. For NWSTG products, the product identification begins at the 24th character of the first product buffer. This is part of the WMO header which follows the CCB header. If a match is found then a file name is generated which consists of the full directory name from the patterns configuration file, part of the product identification from the (WMO) header, and a time stamp with millisecond resolution. Sometimes the product identification from the WMO header contains embedded blanks. These are first stripped before generating the file name.
For GOES, NWSTG/TEXT, NWSTG/POINT, and NWSTG/OTHER products, the time stamps are from when the acqserver process started receiving the product. For NWSTG/GRID and NWSTG/GRAPHIC products, part of the time stamp is taken from the WMO header, specifically the day of month and the hour. The rest of the time stamp is from when the product is received by the acqserver process. The acqserver process uses the WFO-Advanced LogStream facility which allows logging messages of various severity levels to a log file or the screen. If VERBOSE level logging is turned on then each product read by the acqserver process from the CP is identified and those that matched a pattern are displayed with where they will be stored. If a received product matches multiple patterns then each file pathname (storage location) is displayed in the log. A few sanity checks are made for garbled headers or unknown product types. These are logged as well. All products are first completely written to a temporary file location and then renamed to the proper directory. This prevents the decoders from reading a partial product file. Also, if an NWSTG/TEXT or NWSTG/OTHER product cannot be stored because the file system is full, for example, a notification is still sent down the pipeline to the CommsRouter process.The notifications for the NWSTG/TEXT and NWSTG/OTHER products involve sending the whole product contents rather than just a product identification string from the WMO header.
A configurable delay parameter (in milliseconds) is used by the acqserver process to delay the time between disk writes of GOES image product packets. This was introduced to avoid congesting the LAN where the CP and data servers are connected. The delay parameter is read from (and will be re-read on receipt of a HUP signal) the acqTimingParam.txt file.
The acqserver process is broken up into a parent process which waits for connect requests from a client process running on the CP and two child processes. The parent process spawns child processes to handle each data stream (GOES_WEST and NWSTG). The parent process starts by creating a listening socket and then waiting on an accept for the CP client process (acq_client) to connect to it over a "well known" port. The acq_client process on the CP sends a DWB transaction message requesting a direct connection to the acqserver for the purpose of sending data over one of the two data streams. When the acqserver process receives the connect request it forks (with no exec) a child "worker" process to handle reading that particular data stream from the CP. The parent acqserver process goes back to waiting for additional connect requests from the CP. After the CP starts sending over both data streams there will be three acqserver processes running on the data server: the parent process and the two child worker processes.
The regular expression pattern/directory storage location file is named acq_patterns.txt. The timing delay parameter used to avoid network congestion when reading large GOES image files is stored in acqTimingParam.txt. Both configuration files are re-read when the child acqserver processes are sent a Hang Up (HUP) signal.
The radar ingest system for the WSR-88D class I interface is responsible for requesting data from one or more associated Radar Product Generation machines (RPGs), receiving data from the RPGs and making the data available to the D2D workstations. More specifically, this system is responsible for initiating and maintaining connections with multiple RPGs, forwarding data requests from the workstations to the RPGs, receiving and processing data and status messages from the RPGs, and forwarding status messages to the workstations.
The radar ingest system manages three types of requests: the Routine Product Set (RPS) list, the one-time request, and the alert request.
- The RPS list contains the products to be sent routinely to the workstation. The list is sent to the RPG on startup of the system, when the mode of the radar changes, or when a forecaster at the workstation modifies one of the adaptation RPS lists and sends that out to the RPG.
- The one-time request enables a forecaster at the workstation to request that a product be sent on a one-time basis only. The forecaster can also request that the product be sent for more than one volume scan. When the request has been honored, the request is cancelled.
- An alert request is generated at the workstation when the forecaster wants to be notified of an alert condition. When the threshold values specified in the request are exceeded, the RPG sends back an alert message. The forecaster can also request that a product be sent back along with the alert.
The radar data ingest system is composed of two major processes: wfoApi and Radar Server (see Figure6.1). wfoApi makes and maintains connections with the RPGs using software libraries provided by Simpact to manage the transmission and receiving of messages from the RPG. There is one wfoApi process for each associated RPG. The RadarServer provides an interface between wfoApi and the workstations, routing data requests from the workstations to the appropriate wfoApi processes, and forwarding RPG status messages from wfoApi to the workstations. The RadarServer is also responsible for routing incoming data to the main data ingest system where data processing takes place. There is one RadarServer process in the radar data ingest system for handling interactions with associated RPGs.
On startup, wfoApi establishes a connection with the RadarServer process using inter-process communications. The Radar Server then sends out the "current" RPS list which wfoApi formats and sends on to the RPG.
When the RPG sends back a message, wfoApi receives it from the Freeway Call System Application Program Interface (CS-API) in 1024-byte packets. As each packet is received, wfoApi assembles the message based on the message length from the header. When the message is complete, wfoApi stores it in raw files and sends a notification to the Radar Server. If the message is a General Status Message, Alert Adaptation Parameters, an Alert Message, or a Product Request Response, the notification contains the actual message, and the Radar Server sends the message on to be processed for the workstation. If the message is a product, wfoApi sends a "data" notification to the Radar Server, which also sends a "data" notification to the Data Processing Subsystem. Within the Data Processing Subsystem is the RadarStorage process which moves the data from the "raw" data files to "product" directories. A high-level data flow is given in Figure6.1.
Figure6.1
wfoApi
wfoApi is responsible for maintaining the connection with the RPG, and passing information between the RPG and the rest of the data ingest system. wfoApi has a direct interface with the Simpact's Freeway 1000 system using high-level CS-API library functions. The Freeway 1000 system provides serial communications services access to client processes using the Transport Control Protocol/Internet Protocol (TCP/IP) over an Ethernet local area network. In order to handle both normal and high priority data such as alerts, wfoApi establishes two permanent virtual circuits (PVCs). Once wfoApi has connected to the PVCs, data transfer can take place.
wfoApi has a very simple design. It responds asynchronously to external events, and uses tables and configuration files for initialization. wfoApi is started by a parent process called syncComms, the synchronous communications server. syncComms is patterned after Simpact's x25_manager described in the Freeway X.25/HDLC Configuration Guide. On startup, syncComms configures the links, and then passes control to wfoApi.
Some of the asynchronous events that occur are:
- CS_ATTACH_SUCCESS: occurs when wfoApi starts up and attaches to the specified communications server, and the attach has completed successfully. In this state, wfoApi binds the application to the X.25 protocol, establishing the connection.
- CS_BIND_SUCCESS: occurs when the bind completes successfully. When this happens, wfoApi sends a registration message to the Radar Server, followed by a "connection up" message.
- CS_READ_COMPLETE: occurs when the CS-API has completed reading a message from Simpact's Intelligent Communications Processor (ICP) board. The CS API sends the event to wfoApi. When wfoApi gets this event, it reads the data from the appropriate port on the Freeway. When all of the data are read in for a particular message, that message is stored and the Radar Server is notified. If the message is a General Status Message, Alert Adaptation Parameters, Product Request Response or Alert Message, wfoApi sends the message as a notification to the Radar Server.
- CS_LOST_CONN: occurs when either side of the link disconnects. wfoApi sends a "connection down" message to the Radar Server and then exits. On exit, wfoApi cleans up after itself and disconnects.
- CS_WRITE_COMPLETE: occurs when the CS-API detects that a write to an ICP port has completed. On receipt of this event, wfoApi goes back into a wait state.
wfoApi interacts with the RadarServer using IPC class libraries. The classes include:
- SyncCommsClient: handles receipt of the RPS list, One-time request, and Alert request from the RadarServer. Sends out the request.
- RpgRequest: swaps the bytes of the RPS list and the one-time request, and issues cs_write to write the request to the port.
- RpgAlertRequest: swaps the bytes of the alert request, and issues cs_write to write the request to the port.
RadarServer
The Radar Server is responsible for routing messages, and needs to do this efficiently. The server was designed to do very little I/O, and uses the WFO IPC classes in asynchronous mode, in an effort to achieve maximum throughput. The server was modeled after the CommsRouter process (part of the Data Processing Subsystem), which has similar requirements.
The Radar Server was designed using object-oriented analysis and design techniques described in Object-Oriented Modeling and Design by Rumbaugh, Blaha, Premerlani, Eddy, and Lorensen. The major classes that resulted from the analysis, and the event traces that show interaction between classes are discussed below.
Some of the major classes are:
- RadarServerClient: a subclass of the RadarServer class. Responsible for processing all IPC messages the Radar Server receives. Maintains a list of wfoApi processes that have registered.
- RadarStatus: maintains current state of a radar RPG. Radar ID, radar name, connection status, current operational mode, current volume coverage pattern, current RPS list, current One-Time Request list, current Alert Request list.
- ProductRequestList: provides functions for creating a list of products to request from the RPG. Sends the list to a specified target. Writes the list to disk in editable ASCII format, reads list from disk. Send to multiple radars.
- CurrentRpsList: subclass of ProductRequestList. List of products that have been requested from the RPG. Replaces the current list with a default list from disk, or with a list received from a workstation. Formats the list for sending to an RPG, and sends the formatted list to a specified wfoApi process.
- OneTimeReqList: subclass of ProductRequestList. List of one-time requests that have been sent to the RPG.
- AlertRequest: handles the receipt of an Alert request from the workstation.
The Radar Server takes action in response to receiving an IPC message from either a wfoApi process or a workstation process. The messages received from the wfoApi processes and the subsequent actions are listed here:
- Registration message received: RadarServerClient adds RadarStatus object to list of registered wfoApi processes.
- Connection up message received: CurrentRpsList reads current RPS list from disk, and sends list to the wfoApi process. RadarStatus sends "connection up" notification to workstations.
- Connection down message received: RadarStatus removes the OneTimeReqList and sends a "connection down" notification to the workstation.
- General Status Message (GSM) received: If the operational mode has changed, the CurrentRpsList reads a default RPS list for the new mode from disk, and sends this list to the wfoApi process. RadarStatus forwards a GSM notification to the workstations.
- Product notification message received: RadarServerClient forwards the notification to the data processing subsystem.
- Product Request Response (PRR) message received: OneTimeReqList removes the request from the list. RadarServerClient sends PRR notification to workstations.
- Alert Adaptation Parameters (AAP) message received: RadarServerClient sends AAP notification to workstations.
The messages received from the workstations and subsequent actions are listed here:
- RPS list message received: CurrentRpsList stores list on disk, formats the list for the RPG, and sends list to appropriate wfoApi processes.
- One-time request message received: OneTimeReqList adds request to the list, formats the request for the RPG, and sends the request to the appropriate wfoApi processes.
- Alert request message received: AlertReqMsg writes request to file (for retrieval later if the system goes down), formats the request for the RPG, and sends the request to the appropriate wfoApi processes.
The radar ingest system accesses static information from various tables and files. This feature provides flexibility and expansibility. Some of the primary tables and files are listed below under the process heading.
syncComms
- fw1000_pvc(n).setup: This is a "command" file read by the syncComms server to configure X.25 on the freeway ICP for port n.
wfoApi
- cs_config(n): This configuration file is read by wfoApi to provide specific run-time parameters to the CS-API. These parameters are used by the CS-API to associate X.25 circuits with low-level session names. This information is used to access lower level configuration files.
- freeway.config: contains the maximum block size read by wfoApi. It is currently 1024 but could go as high as 8192 bytes.
- portInfo.txt: contains all the ports dedicated to associated radars. It contains such information as port number, board number, radar name, and radar ID.
- productId.txt: contains the product IDs and product types of all possible radar products. An example of a product type is "Graphic."
No static tables or files are associated with the RadarServer process.
The dial-out interface provides the forecaster at the workstation with the ability to send out a request for a product on a one-time basis to a non-associated RPG. When the radar ingest system receives the request, it dials the appropriate radar over one of the available dial ports. Once the connection is established, the request is sent out. The connection remains in effect until all requests to that radar are honored for that RPG. The interface can also handle multiple one-time requests going out to the same radar as long as the connection is up. One caveat here is that there are adaptation parameters on the RPG which limit the number of outstanding requests to five (5).
There can be multiple connections open for multiple radars, but only one connection for any single radar.
The dial-out (Class II) system is very similar to the Class I system. There are two major processes, the Dial Radar and the Dial Server. DialRadar makes and maintains connections to the RPG using the CS-API library routines provided by Simpact to manage the transmission and receipt of messages to and from the RPG. DialServer is patterned after the Radar Server. It provides an interface between DialRadar and the workstation, routing data requests from the workstation to DialRadar, and forwarding RPG status messages from the DialRadar process to the workstation.
The association between DialRadar and DialServer differs from that of wfoApi and Radar Server, in that DialRadar is a child of DialServer.
On receipt of a one-time request from the workstation, DialServer starts up DialRadar for the specified radar, only if a dial connection to that radar is not yet established. DialRadar dials the RPG, sends out a one-time request, and waits for a message from either DialServer or the RPG. The receipt of messages from the RPG is handled in the same way as in the Class I system. As each message is received, DialRadar assembles the packets into a complete message based on the message length from the header.
One of the first messages received from the RPG is the product list which contains all products available on the RPG. The RPG also sends back a General Status Message (GSM) and either the requested product or a Product Request Response message (PRR). DialRadar stores the messages into "raw" data files and sends a notification to the Radar Server. The same procedure is followed as in the Class I interface. A high-level data flow is given in Figure6.2.
Figure6.2
DialServer
DialServer is responsible for routing messages and starting up DialRadar. DialServer uses WFO-Advanced IPC classes in asynchronous mode in an effort to achieve multiple throughput. It is patterned after RadarServer and uses similar classes as defined below:
- DialServerClient: subclass of RadarReceiver class. It is responsible for processing all IPC messages the DialServer receives. It maintains a list of Dial Radar processes that have registered.
- DialRadarStatus: maintains current state of the RPG. The state information consists of radar ID, radar name, connection status, current one-time request list.
- One-timeRequestList: subclass of ProductRequestList. It maintains a list of the one-time requests that have been sent out to an RPG.
- Prr: decodes the Product Request Response message.
DialServer takes action in response to receiving an IPC message from either a DialRadar process or a workstation process. Messages received from DialRadar and the subsequent actions are listed here:
- Registration message received: DialServerClient adds DialRadarStatus object to the list of registered DialRadar processes.
- Connection Up message received: OneTimeRequestList extracts the repeat count from the request and adds the request to the one-time request list. It then sends the request to the Dial Radar process. DialRadarStatus sends a "connection up" message to the workstations.
- Connection Down message received: DialRadarStatus calls on OneTimeRequestList to remove all requests from the list, and sends a "connection down" message to the workstations. DialRadarStatus also removes DialRadarStatus object for DialRadar from the list of registered DialRadar processes.
- Product Request Response message received: OneTimeRequestList checks the PRR to see if the product will be available in the next volume scan. If not, then the product is removed from the one-time request list. If the list is empty, DialRadarStatus sends a "disconnect" message to the Dial Radar process. DialRadarStatus sends a PRR notification to the workstations.
- Product Notification message received: OneTimeRequestList decrements the repeat count for that request. If the repeat count is zero, it removes the request from the list. If the list is empty then DialRadarStatus sends a "disconnect" message to DialRadar. If the repeat count is not zero it updates the repeat count. DialRadarStatus sends a "data" notification to the workstations.
- General Status Message or Product List message received: DialRadarStatus sends the appropriate notification to the workstations.
The messages received from the workstations and the subsequent actions taken are listed here:
- One-time request message received: DialServerClient checks whether the requested radar is in the list of registered DialRadar processes. If not, DialServerClient forks a child process, DialRadar, for that radar. Once the connection is established, DialRadarStatus calls on the OneTimeRequestList class to add the request to the one-time request list and to extract the repeat count. DialRadarStatus then sends the request to DialRadar.
DialRadar
DialRadar is responsible for establishing and maintaining a dial-out connection with the RPG, and for passing information between the RPG and the rest of the radar data ingest system. As in the case of wfoApi, DialRadar establishes a direct interface to the Freeway as discussed in Section6.3.1.3. Unlike a dedicated line, the dial-out connection is temporary and remains in effect only as long as there are outstanding requests.
On startup, DialRadar does the following:
- sends a registration message to the Dial Server;
- gets the telephone number and password from the configuration file using the radar name passed to it by the Dial Server;
- gets an available port and tags that port as "busy;"
- runs the x25_manager as a child process to configure the link as HDLC;
- dials the RPG. On successful connection, DialRadar sends a "connection up" message to the Dial Server, and runs the X.25 manager as a child process to reconfigure the link as a PVC for data transfer. If connection is unsuccessful, DialRadar sends a "connection down" message to the Dial Server.
The messages received by DialRadar from the RPG and the Dial Server, and the subsequent actions, are listed below:
- One-time request received from the DialServer: DialRadar formats the one-time request, sends it out and waits for a message from either the RPG or the Dial Server.
- ProductList message, GSM, or PRR from the RPG: DialRadar stores the product in the "raw" data file and sends the message to DialServer.
- Product message from the RPG: DialRadar stores the product in the "raw" data file and sends a product notification to DialServer.
- Disconnect message from DialServer: DialRadar disconnects from the RPG, tags the port as "free" and sends a "connection down" message to DialServer.
All DialRadar actions are asynchronous. As an event occurs, the event is processed and DialRadar goes into a "wait" state. Some of the asynchronous events are listed below:
- CS_ATTACH_SUCCESS: The CS-API sends this event to the client, DialRadar, on completion of a successful cs_attach. cs_attach attaches the client to the specified communications server.
- CS_BIND_SUCCESS: DialRadar issued a cs_bind which completed successfully. cs_bind binds the client to the HDLC or X.25 protocol.
- CS_CONNECT_SUCCESS. DialRadar issued a cs_connect which completed successfully. cs_connect enables the client to establish an HDLC connection to the remote modem (DTE).
- CS_WRITE_COMPLETE: DialRadar issues cs_writes to send the password and telephone number to the modem, and to send out information to the RPG. On successful completion, the CS_API sends this event to the client process.
DialRadar
- fw1000_hdlc.setup(n): This is a "command" file read by the x25_manager to configure HDLC on the Freeway ICP for port n.
- fw1000_pvc(n).setup: This is a "command" file read by the x25_manager to configure X.25 on the Freeway ICP for port n.
- cs_config(n): This configuration file is read by DialRadar to provide specific run-time parameters to the CS-API. These parameters are used by the CS-API to associate X.25 circuits with low-level session names. This information is used to access lower level configuration files.
- freeway.config: This file contains the maximum block size read by DialRadar. It is currently 1024 but could go as high as 8192 bytes.
- dialPorts.txt: contains information on all the dial ports available on all ICPs. It contains such information as port number and board number.
- dialRadars.txt: contains all the radars containing dial ports. The information consists of radar name, radar ID, telephone number, and password.
- productId.txt: contains the product IDs and product types of all possible radar products. An example of a product type is "Graphic."
Automation of field offices' interactions with local data observation systems, spotter networks, cooperative observers, and members of the local decision-making community is recognized as vital to the success of the NWS modernization. This automation is required (1) to acquire observational data to complement the basic federal weather observing systems and (2) to disseminate warning information to key decision makers in local and state communities.(1)
The proposed LDAD functionality for WFO-Advanced includes a suite of PC hardware outside the AWIPS network at the WFO. This hardware acquires and disseminates data to the general public while preserving the security efforts necessary for the AWIPS network.
LDAD has several intended users in the WFO network. The forecaster depends on these data for reliable forecasting and verification within the local scale. The local analysis extensively uses the local data for results. Application programs (such as the WHFS applications) use these data for further analysis.
In order to accommodate these users, FSL is investigating a generic database for storage of these data sets. Currently, the data are stored in flat "plot" files, unusable by any users but the WFO-Advanced D2D display. The concept is to have a generic netCDF file in which any local data point can be stored. This file would have enough self-describing information for identification by any end-user.
Currently, the Denver system includes two local data sets: a set of surface sensors maintained by the Colorado Department of Transportation (CDoT), and a set of sensors gathering hydrological information maintained by the Denver Urban Drainage and Flood Control District (UDFCD).
The Colorado Department of Transportation (CDoT), with Surface Systems, Inc. (SSI), operates a network of approximately 38 Remote Sensing Units (RSUs). Each RSU collects data from 5 sensors 1 surface and four sub-surface that measure such phenomena as air temperature, precipitation, dew point, etc. The CDoT polls the RSUs every 15 minutes and transmits the data to FSL's Community Server.
Every fifteen minutes, a process running on the WFO-Advanced data server copies the current CDoT file via FTP from the Community Server to WFO-Advanced, where the file is stored for later decoding. The decoded data are stored to disk in both flat file and netCDF formats. The flat file is later accessed by the workstation and translated on the fly for display on the workstation.
The CDoT data are copied to the Community Server from the CDoT every 15 minutes. From here the data are copied to WFO-Advanced and stored in $FXA_DATA/async/cdot. A notification is sent to the Comms Router of data arrival. The Comms Router sends on a notification to the Data Controller which then sends on the data to the CdotDecoder. The decoder translates the data from ingest format into meteorologically meaningful values and stores the decoded data in both flat file and netCDF file formats.
- ftpCdot: started via a cron job which runs every fifteen minutes starting at 5 minutes after the hour. ftpCdot copies the data from the Community Server, stores the data to disk, and notifies the Comms Router.
- Comms Router: sends a notification to the Data controller.
- Data Controller: passes on the notification to the CdotDecoder.
- CdotDecoder: a child process started by the DataController. On receipt of a notification from the Data Controller, the decoder reads the data file from $FXA_DATA/async/cdot and decodes the data. It reads its station information file and generates a report that is stored both in flat file and netCDF file formats. Both of these files are hourly files based on observation time.
The decoder reads station information from file $FXA_HOME/data/cdotStationInfo.txt. This file contains the station ID, latitude, longitude, and station description.
The ALERT data are hydrometeorological observations from remote data collection platforms (DCPs). These DCPs are connected to a base station, at the Denver Urban Drainage and Flood Control District (UDFCD), via radio communications.
The ALERT data are copied to WFO-Advanced by an FTP cron job from the Community Server. This scheduled copy is synchronized with the receipt of data from the Denver UDFCD to the Community Server. The Community Server receives these data by a land line link (modem), and after a short delay the data are relayed to WFO-Advanced.
The AlertDecoder is started up by a DataController process. When started, the AlertDecoder registers a pattern string and a callback function with the DataController. When the DataController receives a message containing the AlertDecoder pattern string, this message is passed to the callback function, instantiating the AlertDecoder process.
Upon successful completion and transfer of ALERT data files from the Community Server, the FTP process sends an ALERT pattern notification to the DataController. (See Section7.6.2.2 for more information on data arrival notification messages.)This IPC notification causes the AlertDecoder to read each of the received data files, decode their content, then store the data. These decoded observations are also "serialized" encoded in a platform-independent manner and sent via an IPC message to the SHEF encoder process, where the message is "quantized" decoded from the "serialized" message and prepared for ingest into WHFS's hydrologic data base.
The ALERT data received from the Community Server are parceled to three files: meteorological non-precipitation weather observations, precipitation observations, and hydrological observations. These files are stored in the $FXA_DATA/async/alert directory while awaiting decoding. As each of these data files is processed, its data contents are stored in LDAD plot files (a display-ready format used by WFO-Advanced) and ALERT netCDF data file (containing a complete data record). When each file is successfully decoded it is unlinked. If there is a failure in the decoding process, the file is moved to the $FXA_DATA/async/alert/badAlert directory to await evaluation of the decoder failure.
A cron job FTPs data periodically to WFO-Advanced from the Community Server and sends a data arrival notification message to a DataController. When the AlertDecoder was started by a DataController, the decoder registered a pattern string to have specific data sets delivered to it. When this data arrival message pattern string is received by the DataController, the message is forwarded to the decoder. Upon receipt of the forwarded message, the AlertDecoder reads the data files in $FXA_DATA/async/alert and decodes them, storing their data in LDAD plotfiles and netCDF data files. The AlertDecoder also extracts the relevant hydrologically significant data from the decoded data and forwards this using an IPC data message to the SHEF encoder, where these data are SHEF encoded for ingest into the WHFS hydrologic database.
The names of the un-decoded ALERT observations received from the Community Server are alert_wx.dat (the meteorological non-precipitation data observations),alert_r5.dat (the incremental precipitation observations), and alert_wl.dat (the hydrologic observations). All are stored in $FXA_DATA/async/alert.
The AlertDecoder uses the AlertStationInfo.txt file to provide the necessary location (latitude, longitude, and elevation) data and station name details that are stored in the LDAD plotfile and netCDF data files along with the decoded observations.
The decoded data are stored in the $FXA_DATA/point/localdata directory where the plotfiles are named by the date and hour of data that are contained in the file. Similarly-named netCDF files are stored in $FXA_DATA/point/alert.
Footnotes
- (1)
- User Language Requirements Document (ULRD) for the Local Data Acquisition and Dissemination (LDAD) Function of AWIPS, May 1994, NWS Internal Document.
Table of Contents 
Next Chapter
This document is maintained by Joe
Wakefield. Last updated 9 Jan 97.