MADIS Overview

Introduction

patricia.a.miller@noaa.gov

http://www-sdd.fsl.noaa.gov/MADIS

Ingest

Meteorological Aviation Reports (METARs)
Surface Aviation Observations (SAOs) *
Surface maritime reports (including Buoy, Ship, and Coastal-Marine Automated Network [C-MAN])
Modernized NWS Cooperative Observer (COOP-M) Program *
Surface Mesonets * from...
Radiosondes
NOAA Profiler Network (NPN)
Cooperating Agency Profiler (CAP) *
Automated aircraft reports and profiles (including MDCRS, ACARS *, AMDAR *, and TAMDAR * observations)
Microwave Radiometers *
GOES 3-h operational winds
GOES 1-h experimental winds *

* Indicates datasets not currently available on NWS AWIPS via NOAAPORT

MADIS is also currently serving as the modernized COOP (COOP-M) Central Facility by ingesting raw COOP-M observations from five different datastreams, integrating them, applying automated QC procedures, and distributing them to interested parties and organizations. Automated monitoring tools designed to locate communications outages at individual COOP-M stations are also supported by MADIS.

Figure 1. MADIS observations available over the east coast of the U.S. on April 29, 2004.
a - left) All MADIS observations at 00Z. Surface observations are in green, aircraft reports in blue, and satellite winds in black. Profiler locations are indicated by a "P" and radiosonde locations with "X".
b - right) MADIS surface observations at 12Z. Standard METAR and maritime data are in red. Locations of MADIS mesonet and COOP-M stations are indicated by blue "X"s.

Integration

Raw data files are received from many different data providers in many different data formats, and contain observations in various units from stations reporting from various time zones. MADIS ingests these data files, combines the observations from non-NOAA data providers, and integrates them with NOAA datasets by encoding all of the data into a uniform format and converting all of the observations to standard observation units and time stamps.

Figure 2. MADIS mesonet stations in the North-American domain on May 26, 2004.

Automated Quality Control

Two categories of QC checks, static and dynamic, are implemented for each observation type. The static QC checks are single-station, single-time checks which, as such, are unaware of the previous and current meteorological or hydrologic situation described by other observations and grids. Checks falling into this category include validity, internal consistency, and vertical consistency. Although useful for locating extreme outliers in the observational database, the static checks can have difficulty with statistically reasonable, but invalid data.

To address these difficulties, MADIS also implements dynamic checks which refine the QC information by taking advantage of other available hydrometeorological information.

Examples of dynamic QC checks include position consistency, temporal consistency, and spatial consistency. QC checks are run on a subhourly basis to guarantee the timeliness of the information to MADIS users. Static QC checks, for example, are applied every five minutes to newly arrived surface observations, while the spatial consistency check is run every 15 minutes.

The spatial consistency check is performed using an Optimal Interpolation (OI) technique. At each observation location, the difference between the measured value and the value analyzed by OI is computed. If the magnitude of the difference is small, the observation agrees with its neighbors and is considered correct. If, however, the difference is large, either the observation being checked or one of the observations used in the analysis is bad.

To determine which is the case, a reanalysis to the observation location is performed by eliminating one neighboring observation at a time. If successively eliminating each neighbor does not produce an analysis that agrees with the target observation (the observation being checked), the observation is flagged as bad. If eliminating one of the neighboring observations produces an analysis that agrees with the target observation, then the target observation is flagged as "good" and the neighbor is flagged as "suspect."

Suspect observations are not used in subsequent OI analyses. The reanalysis step is particularly important when combining data from well-maintained and well-sited observation systems with data from less advantaged observing systems.

Figure 3. Graphic illustration of reanalysis procedure used in the MADIS spatial consistency check to determine if the target observation is bad or if one of the observations used in the QC analysis is bad. The reanalysis procedure is implemented only if the difference between the target observation and the analysis is greater than a specified threshold.

In addition, MADIS provides the capability for human interaction with, and override of, the automated QC results by maintaining "reject" and "accept" lists for each dataset. Observations placed on the reject list are always flagged as "bad" within the MADIS QC database, while observations placed on the accept list are flagged as "good."

Quality Control Monitoring for Surface Datasets

The monitoring statistics are computed on an hourly, daily, weekly, and monthly basis for individual NOAA and non-NOAA surface networks (e.g. ASOS, COOP-M, MesoWest, etc.) and have proven to be very useful in locating persistent observational biases and hardware failures. Text versions of the statistics are made available to data providers as part of the MADIS data distribution and web services.

The Browser is an AWIPS-based display application that serves as the user interface to the MADIS surface QC database by implementing an interactive text and graphics display system designed to provide visualization of the QC information, and also to provider easy access to the MADIS subjective intervention capabilities. The Browser is configured at FSL for use in monitoring NOAA surface networks, as well as monitoring non-NOAA mesonets.

Users of the QCMS Browser can select all or portions of the QC information provided in the MADIS database and display the information as plan view and/or time series plots, or can choose to display the information in tabular form on the user interface. The Browser provides easy access to the MADIS database and QC information for the purposes of:

monitoring station performance;
locating persistent biases or failures in surface observations; and
evaluating observation/QC accuracy.

The Browser also interacts with the MADIS subjective intervention capabilities to allow individual variables at select stations to easily be either removed from or placed onto the reject or accept lists. MADIS automated QC and station monitoring procedures are not affected by subjective intervention lists, with the sole exception that observations on the reject list will be labeled as "suspect" and not used to check the spatial consistency of neighboring observations. This allows for the continued monitoring of the stations contained in the lists. For example, a station with wind observations that fail the QC checks a large percentage of the time may be added to the reject list. However, once the observation failure rate at the station falls back to near zero (possibly due to an anemometer that has been repaired), the station will likely be deleted from the list.

Current resources at FSL do not allow for continuous human monitoring of surface meteorological observations. Instead the subjective intervention lists are maintained in response to data issues raised by users or noticed by the MADIS developers. In any case, the QCMS Browser has been found to be a useful tool for monitoring surface observations and QC, and should be considered for use in NOAA operational monitoring centers. Plans are also in the works to install the Browser at each NWS forecast office as part of their operational AWIPS systems.

Figure 4. QCMS Browser-produced AWIPS displays detailing the detection and correction of a persistent bias in the sea-level pressure observations reported by an ASOS station in Ypsilanti, MI. a) top, Plan view display of the weekly percentage of QC failures indicating a 100% failure for the Ypsilanti, MI, sea-level pressure observations before NWS personnel were alerted to the problem. Percentage failures above 95% are highlighted in red. b) below, Time series display of the daily RMS (blue) and mean (red) errors for the observations, and the percentage failure (green) over the days both before and after NWS personnel corrected the problem. The QC information before the correction indicated persistent RMS and mean errors of approximately 2.1 mb and a continuous failure rate near 100%. After the observations were corrected, both the errors and the percentage failure fell to zero.

Output Data Files and Software Support

The output files for all MADIS datasets are available in uniform formats with uniform quality control structures within the data files. Software support is also provided for the datasets through the use of an Application Program Interface (API) that provides users with easy access to the data and QC information. The API allows each user to specify station and observation types, as well as QC choices, and domain and time boundaries.

Many of the implementation details that arise in data ingest programs are automatically performed by the API, greatly simplifying user access to the disparate datasets, and effectively integrating the database by allowing, for example, users to access many different types of surface observations (e.g. ASOS, COOP-M, maritime, and non-NOAA mesonets) through a single interface. Users of the MADIS API, can also, for example, choose to have their wind data automatically rotated to a specified grid projection, and/or choose to have mandatory and significant levels from radiosonde data interleaved, sorted by descending pressure, and corrected for hydrostatic consistency.

For users who do not want to write their own programs to use the API, but would like easy access to the data, utility programs for each dataset are included in the API package. These programs can be used to read station information, observations, and QC information for a single time, and dump these to an output text file.

The operation of each program is controlled by a text parameter file that allows the user to exercise all of the options included in the MADIS API. The programs can be run as needed to access any MADIS data stored on the user's system, or can be run as time-scheduled tasks to get data keyed to the current time.

AWIPS Support

Also, where applicable (such as for GPSMET integrated precipitable water vapor), the documentation will explain how to customize AWIPS to display additional variables that are not part of the baseline AWIPS system.

In addition to the LDAD data, several of the MADIS datasets are supplementary to what's currently available to AWIPS via NOAAPORT. Customization packages (including instructions and the necessary files to be used in the AWIPS localization tasks) are provided.

Metadata Updates

Station tables are updated for all of the datasets as needed, including weekly automated updates for the mesonet data. In addition to the station table maintenance, additional metadata such as site photos, are collected when available.

Figure 5. AWIPS Cooperating Agency Profiler (CAP) time-height plot for the Ft. Meade profiler provided by the Maryland Department of the Environment. MADIS supports NWS AWIPS users by providing both real-time access to CAP data and instructions for how to ingest and display the data on AWIPS.

Figure 6. Photograph indicating the northern view from the Alton, IA station in the Iowa Department of Transportation mesonet. Photographs of the northern, eastern, southern, and western views for each station in the mesonet were provided by the Iowa Department of Transportation and are kept on the MADIS web server for reference by users of the mesonet data.

User Support

MADIS developers and support staff are available via telephone and/or e-mail to assist users of the MADIS datasets and API. FSL staff, for example, frequently assist NWS forecast office staff with setting up AWIPS ingest and display of MADIS datasets.

FSL Central Facility Support

the Data Systems Group which acquires and processes meteorological datasets;
the High-Performance Computing Group which operates and maintains one of the world's fastest supercomputers; and
the Systems Support Group which provides real-time computer operator support for the FSL Central Facility.

In addition to the functional groups, ITS staff also provide the network management, computer systems administration, and computer security services required by the MADIS project.

Figure 7. Web-based monitoring system used by the FSL operators to assess the availability of the MADIS datasets. Green "state" lights indicate that datasets are available; red indicate missing data. In this case, GOES High-Density Winds (HDW) were missing due to a server failure at NESDIS.

Processing Environment

The MADIS data processing is performed on a system of 21 computers using Intel processors and the Linux operating system, many of which are configured into "high-availability" (HA) pairs. The HA pairs are used to increase reliability by providing redundant hot backups for the key processing components. A failure of the primary computer will be immediately detected by the secondary computer, which will then automatically start distributing its outputs to the downstream computers.

Figure 8. Schematic depiction of the MADIS processing environment. FSL is currently in the process of installing "high-availability" pairs for each MADIS processor to help increase the reliability of the MADIS datasets.

Distribution

Since some of the data are proprietary, different distribution categories have been set up to handle restricting these datasets, which include some of the mesonets, and the automated aircraft data. In general, no restrictions apply to government agencies supporting forecasting operations, or to research and educational organizations or institutions.

To accommodate the different distribution categories, all distribution mechanisms include authentication, with different levels of authentication required for the more restricted datasets (e.g., the Internet address of the user's computer is validated in addition to requiring a password-based ftp account).

Additional services are provided so that all MADIS data files are directly compatible with AWIPS displays and applications without the use of the MADIS API software, and without the need to change AWIPS software to handle new datasets.

Data Storage

In addition to the real-time data available from the MADIS servers, on-line access is available to saved real-time data going back, for most datasets, to July 1, 2001. These data can be accessed either via ftp or by OPeNDAP clients. The MADIS API package includes OPeNDAP clients that can be used to access each dataset. To provide a backup in case of disk failure on the servers, the FSL Central Facility mass store system also contains a complete set of MADIS data files.

Data Outage Monitoring and E-Mail Notifications

Operators also maintain logs of data outages for internal communication and tracking purposes, and take corrective action in response to failed real-time systems based on documentation created for each system or subsystem. The operators are also responsible for maintaining the troubleshooting documentation and other pertinent informational and training materials.

User Response

From Ron Sznaider, Vice President for Business Development at Meteorlogix Corporation:
From the U.S. Weather Research Program's Advanced Operational Nowcasting System Working Group (who acknowledged that the establishment of a national surface station mesonetwork was their highest priority for new or enhanced observing systems):
From Jeffrey Medlin, Science and Operations Officer, NWS Mobile, Alabama Weather Forecast Office:
From Dr. Pat Welsh, Science and Operations Officer, NWS Jacksonville, Florida Weather Forecast Office:
From Dr. Jan F. Dutton, Director of Weather Services, AWS Convergence Technologies, Inc.

Short-Term Plans

FSL plans to support the Great Lakes Flight Experiment, scheduled for the fall of 2004, through its MADIS system by ingesting, integrating, quality controlling, and distributing Tropospheric Airborne Meteorological Data Reporting System (TAMDAR) sensor data to numerous experiment participants, including NCEP and NWS forecast offices. Support for additional datasets, such Polar orbiting satellite observations and observations from up to 25 new surface mesonets is also planned for the 2004 – 2005 time frame, as well as support for new distribution capabilities, such as XML-based output files. A "data recovery system" to process surface observations that were not originally available in real time due to communications outages is also under development.

Figure 9. Mesaba Airline SAAB-340 Flight routes planned for the Great Lakes Flight Experiment. Mesaba plans to install TAMDAR units on 64 SAAB-340 aircraft (routes in green) to support the Experiment. Raw data from the units will be ingested, integrated, quality controlled, and distributed by the MADIS system at FSL.