LDAD System Manager's Manual

Appendix A - How to Add a New Data Source

This appendix describes adding a new LDAD data source. Files describing the data format (.desc) and station info (Station.txt) need to be created, and a preprocessor may be required. The rest of the work is accomplished using the LDAD monitor Admin interface.

Set up metadata and preprocessor

• If you have a choice, decide on a name for your data files. (In many cases, the name is dictated by your provider.) It is important to choose unique names for these files, because the listener process, which pulls the data from ls1:/data/Incoming to ds1:/data/LDAD/tmp, will use the name as a case-insensitive string to match against all of its entries.
• If you are dealing with station data, create .desc and Station.txt files and place them on ds1 and ls1 in ~ldad/data. Be careful to include the right number of fields (vertical bars) in the Station.txt file; the decoder is picky about that.
• Write a preprocessor. (While listener will move the file from $LDAD_DATA/tmp to $LDAD_DATA/Raw for decoding if no preprocessor is specified, even if that's all you want to do it's wise to write a very simple moveYourfile.pl script, to make it clear.) The .desc file describes the format of the data after preprocessing, and one header line may be present. Additional descriptive header lines may be in the file if preceded with a #. You should test your preprocessor by hand, to verify that the output file is as described in the .desc file.

  A set of useful Perl functions is provided in LdadUtils.pl. See Appendix C for a description of these utility functions.

  If you wish to store the message in the text database, you will need to put a file in $LDAD_DATA/Text to make it available to the text storage routine. The standard approach is to put the raw message into the text database. If no preprocessor is otherwise needed, you can write a very simple script, such as this (moveRaws.pl):

    #!/usr/local/perl/bin/perl -w
    unshift (@INC, $ENV{'LDAD_INTERNAL_BIN'});
    require "LdadUtils.pl";
    &LdadUtils::moveFile ("RAWS");
    1;
  

  moveFile puts a copy in both /Raw and /Text. Routines copyToText and copyToRaw are also available in LdadUtils.pl to put the file in /Text and /Raw only, respectively. You would use these, for example, to put the original file in the text database, while sending the preprocessed file to the decoder. In either case, you need to call pingLdadRouter to alert the system that the file is in place. Here is an example:

    my($textFile) = &LdadUtils::copyToText ($file);
    &LdadUtils::pingLdadRouter ($textFile);
  

  If you are reformatting the data, it's best to write first to a scratch file, then move it to /Raw and /Text. Otherwise, since the decoder processes all files it finds, it's possible that it will see your file before you've finished working on it. So use a structure like this:

    $tmpFile =  "$ENV{'LDAD_INTERNAL_HOME'}/ldadtmp/tmp$$";
    open(obout, ">$tmpFile");
    # do your thing, writing to obout
    &LdadUtils::LDADcopy($tmpFile,$ARGV[0]);
    unlink ($tmpFile);
  

Modify the various tables via the Administration interface

• Add an entry to LDADinfo.txt ("LDAD Conf. Table" on the Admininstration page). Using existing entries as a model, put the root name of the .desc file in column 1, and the root name of the Station.txt file in column 2. Although plot files are no longer created, you must put something in the plot file key column - a 0 will do. Be sure to specify the correct storage directory.
  If you are simply passing data through the firewall, you won't have any .desc or Station.txt files, nor will you need entries in the storage columns of LDADinfo.txt. In either case, the entry in column one is used by listener to match against the arriving file name to decide which script to call. Again, please note that this is case-insensitive. Further, if there are redundant entries (e.g., AVN and avn), the last-occuring one will be used. Caution: Take care to ensure that your column-one entries are unique. If necessary, you may either have to persuade a provider to change the file name, or rename the file on your ls1 before writing it to /data/Incoming.
• Add entries to the Edit section of ldadAdmin.conf ("Administration" in the Configure Monitor Tables section of the Admininstraion interface) to allow you to view and modify the description and station files.
• Add provider information to LDADProvider.txt ("LDAD Provider List"). This will show up when you click on the link on the acquisition monitor page.
• Add an entry to the Check section of the Data Acquisition configuration file (via link in the Configure Monitor Tables section of the Administration interface). Note that the data key here should match the entry you made in LDADProvider.txt. Change the time-out values: times less than "green" will show green, those between "green" and "blue" will show blue, between "blue" and "yellow" show yellow, and greater than "yellow" shows red. The "red" time-out doesn't matter; it just needs to be greater than "yellow."
• If necessary, edit LdadPatterns.txt. Select "ingest data patterns" from the Edit section of the Admin page. The default patterns are fairly generic, and probably will accommodate your data.
• If desired, add an ID for storing the text in the database. Select "Text storage control" in the Edit section.

Restart ingest, decoding, and storage

After LDADinfo is changed, you must restart listener to read it and set up the preprocessor accordingly. You'll also need to restart the decode/store suite to read the new patterns and text storage files.

1. Stop the decode/store suite (click "LDAD Decode/Store" on the Stop menu).
2. Click the "All LDAD Processes" link in the Start/Restart section at the top of the Administration page. This will also start the decode/store suite, since it's not running. (This process takes about two minutes.)

Verify the results

• Once you have tested the preprocessor and restarted ingest, look at the log files to see your file pass through the system. After copying a data file into ls1:/data/Incoming, you should see it show up in the listener (it's a "do_G_Msg" log entry; note that you'll see both rcp and preprocessor records), CommsRouter, and DataController logs on ds1. It will also show in the routerLdadDecoder log, and the decoded file will appear in /data/fxa/LDAD/decoded. If the data are successfully stored, you'll find them in /data/fxa/LDAD/mesonet/netCDF (or hydro, manual, upperAir); use /usr/local/netcdf/bin/ncdump to see how they've been stored.
• As fxa on the ds, run ~ldad/bin/relocalizeForLDAD.sh, and restart to see the plots.