Investigation of an AWIPS Distributed Data Architecture

Michael Biere, Darien Davis, Carl Bullock, Herb Grote
13 May 2003

The Problem

The current AWIPS data delivery architecture "pushes" data to every Weather Forecast Office (WFO) via the NWS SBN. All national data sets are transmitted via satellite and stored locally at each WFO. This satisfied the initial AWIPS software deliverables, but the increased hydromet datasets available tax the current hardware and software architecture.

A more flexible approach is to augment the existing data distribution channel with an "on-demand", request-oriented mechanism

Demonstration Prototype

Investigate community available software using this approach (e.g. Unidata THREDDS/openDAP/DODs netcdf server/client interface ). We will try to reuse existing web technologies and software as appropriate (e.g. XML, caching proxies, etc).
Modify workstation software to add relevant client-side software to pull data from a server including current configuration tables;
Install and configure data server host for the workstation clients;
Investigate methods to augment the event notifications from remote data sites;
Stage test environments;
Report on results.

Enhancements

Assuming the demonstration prototype shows the viability of the distributed data approach, we will want to refactor the client software to add a dynamic "inventory server" component. Our demonstration approach will rely on modifications to existing static table structures. The "inventory server" will have knowledge of network topology and availability, and can allow for multiple possible sources for identical data sets.
In addition, some effort should be conducted to analyze the patterns of data usage by a few, well selected WFO sites. Metrics can be produced showing the most-used data sets and providing input toward optimizing the data distribution using these measurements.
Investigate storing grib-encoded grids in transmission format. A flat-file based structure can be used to store the grib files which can then be decoded "on-the-fly" by the client software.
Investigate caching of remote data sets at the WFO to satisfy repeated requests.