AWIPS Build 4.2 Notes

Operationally, the radar ingest system in AWIPS build 4.2 is somewhat different from the previous builds. The main differences are as follows:

1. The syncComms script restarts wfoApi when wfoApi exits.

2. restartRadar is no longer started as a cron job in ingest.crontab.ds1.

3. restartRadar has been added to the following scripts to restart syncComms and wfoApi:

startIngest.ds1
startRadar

4. The script, x25_startSync'n' has been added to icpReset and configBuf to restart
syncComms and wfoApi when the freeway is reset and/or buffers configured.

5. The pseudo-semaphores /tmp/freewayDown'n' have been removed. Processes now
check whether either icp_reset or configBuf is running.

6. The radar ingest, wfoApi and syncComms are started from
         startIngest.ds1
         startRadar
         icpReset
         configBuf

Operations:
-----------

Stopping wfoApi and syncComms:

1. To stop wfoApi on port 'n', run x25_stop'n'

2. To stop syncComms and wfoApi on port 'n', run stopSync 'n'. Please note the space
between stopSyn and 'n'.

3. To stop wfoApi and syncComms on all ports, and pingFreeway, run stopRadar.

Starting wfoApi and syncComms

1. To restart syncComms on port 'n', run x25_restart'n'.

2. To restart wfoApi and syncComms on all ports, and pingFreeway, run startRadar

Monitoring the Freeway
------------------

pingFreeway has been modified to collect statistics on a configured port. It is currently
not being run since it still needs some work. I have disabled the pingFreeways from running on ds1-fsli and ds1-fsld, but they have not been removed them from startIngest.ds1 nor startRadar. So, if there is an upgrade, these processes will run again, and their behavior will be unpredictable.

System Configuration
--------------------

ds1-fsli: AWIPS build 4.1 from the january98 tree.

ds1-fsld: AWIPS build 4.2 from the august98 tree.

Note: All radar changes have been checked into the august98 tree.

Radar Ingest Configuration
--------------------------

dedicated connection: ds1-fsli is connected to Denver's KFTG RPG at 9.6 kbps on port 3,
ICP 0 of Simpact cpsync-fsli

ds1-fsld is connected to OSF's KCRI RPG at 14.4 kbps on port 1,
ICP 0 of Simpact cpsync-fsld

dial connection: the dial line is connected to cpsync-fsld port1, ICP1.
It is primarliy used for testing, but is available for use and should be
pretty reliable. There are still few small bugs to iron out.

Trouble-Shooting
----------------

ds1-fsli:

The dedicated line is very stable. If on the off chance the radar data should stop coming in, then take the following steps:

1. Check the radar status window on the workstation (or look at
    $FXA_DATA/workFiles/RadarAnnouncer) to determine if there is a problem
    with either the RPG or the RDA. Look at the line "New Prod Status". It
    should be followed by "Reflectivity Velocity Spectrum Width". If it
    followed by "... Unavailable...", then there will be no data. The RPG or RDA status
    lines usually indicate the problem. If all looks fine, then

2. Check the RadarServer log file in $LOG_DIR/yymmdd to check if the RPS list
    was sent out. If not, there will be an indication as to the cause.   Another thing to look
    for is "_source Id has been initialized to -1". This indicates an empty or non-existent
    "pupId.txt" file in $FXA_HOME/data/localizationDataSets/XXX, where XXX is the
    site Id.

3. If the RPS list has been sent out and the status logs do not indicate a problem , then there
is a very good chance the simpact is hung. Please follow the freeway reboot instructions
provided to ensure a smooth recovery:

   a. At a terminal, go to $FXA_HOME/bin and run icpReset'n' where 'n' is either 0 or 1.
       If the reset is successful, you should see text that indicates the
                     "ICP n has been initialized"
                     "Configuring buffers on ICP n"
                     " Buffers initialized..."

If the response is:
"ICP 'n' not reset, so reset manually...."

       run icpReset'n' again. If it fails a second time, then either reset or reboot the freeway
       manually. (hit the "reset" button on the front of the freeway box, or power the freeway
       off/on.

b. If icpReset is successful, then we are back in business. You should start seeing data
again.

c. If you needed to manually reset the freeway, then at the terminal,
run $FXA_HOME/bin/configBuf'n'

where n is the ICP number. This will kill wfoApi, dialRadar and any x25_managers,
and configure the buffers on ICP n.

       Please note: if wfoApi, dialRadar or the x25_manager runs while buffers are being
       configured, the configuration will fail!!! You will see the message:

         "Buffers not configured, retrying..."

       This message will appear 5 times. If you kill whatever process is running (dialRadar.
        wfoApi or the x25_manager), the configuration might succeed. If it does not succeed
        run configBuf'n' again. It should work.

d. If the freeway just does not reboot, here's what to do:

check the /etc/inetd.conf file on ds1-fsld

       Sometimes, this file gets changed. Look for the for the simpact boot directory
      /opt/freeway/boot or /usr/local/freeway/boot. If it is not there, have a Sys Admin
      person add it. Follow the simpact reboot instructions given above.

ds1-fsld:

Dedicated:

The radar data flow on this system is pretty sporadic since the OSF is testing their Build 10 system. Sometimes, the data flows profusely. While at other times it stops completely.
It looks as though the analog line has been pulled and we start seeing a lot of noise on the line. So, my recommendation is to check the status of the data once per day and follow the trouble-shooting directions given above if you run into problems. You will most likely have to run $FXA_HOME/bin/icpReset0 about once every day or two.

Dial:

You can make one-time dial requests from ds1-fsld. The DialServer will start up a dialRadar process when it receives a request. A new feature available in AWIPS build 4.1 and 4.2 is the capability to send multiple requests from the one-time request menu. The requests will be queued until a dial port is available. The dialRadar process remains active until the request is satisfied.

Trouble-shooting the dialing process:

1. When you send a dial request to the radar, messages will appear in the radar status
window as follows:

"dialing kxxx...

"One-time request sent successfully...", or

" Request not sent, retrying..."

This usually indicates the request will eventually go out to the RPG

If you do not have access to a workstation, you can also find the messages in the radar
announcer log file $FXA_DATA/workFiles/RADAR_Announcer.

2. If the request actually gets sent to the RPG, either the product or a Product Request
Response will be returned, and a description will show up in the radar status window
and in the radar annoucer log file.

3. If the request is not sent out, there are a number of reasons:

a. The simpact is hung.

The following message will appear in the radar status window after 1 minute:

"Freeway not responding..."

          Run /opt/freeway/bin/x25_manager mgrConfig'n' where n is the ICP number
          (0 or 1). After the ":", enter the command "monitor {links(m), brief", where m is the
           port number (0, 1, 2 or 3). If there is no response, then follow simpact reboot
           instructions above.

If there is a response, then the simpact is not hung,

b. The modem might be hung

The message "Freeway not responding" appears in the status window

         This happens if the modem is interrupted while trying to dial. Have a Sys Admin
          person check the lights on the front of the modem. If the HI/ON light is on, the
          modem is still on-hook. Power the modem off/on and resubmit the request. If the
          HI/ON light is not on,

    c. The RPG you are dialing might have dialing in disabled. Look at the most recent
        dialRadar log file. If there is a line indicating "...bind failed or rejected...", then try a
        different RPG. If there is no indication of a bind failure, then

    d. The DialServer might have queued your request, but there is no other dialRadar
         process running. This is a bug and only occurs if the dialRadar process is killed
         using "kill -9". The DialServer expects a certain response when the dialRadar
         processdies, and if it does not get that response, it does not clean up properly.

         Look at the DialServer log file. If there is a line that says "adding abc to the waiting
         list", where abc is the radar id, then your request was queued. Kill the DialServer
         using "kill pid" and restart it using "$FXA_HOME/bin/DialServer &". (You can find
         out to what radar abc refers by looking it up in the file
         $FXA_HOME/data/localizationDataSets/FSL/dialRadars.txt).