AWIPS BUILD 5.1.1 MSAS SOFTWARE TESTING GUIDE updated 12/20/00 ---------------------------------------------- CHECKING THAT MSAS IS RUNNING CORRECTLY on AS2 ---------------------------------------------- ------------------ Subhourly QC cycle ------------------ Starting at 3 minutes past the hour, the subhourly QC cycle checks for newly arrived LDAD mesonet observations. Any that have arrived, are QCed immediately with the validity, temporal and internal consistency checks. This process is repeated every 5 min throughout the hour (and into the first 15 min of the following hour). To check that it's running correctly: 1. Check the final netCDF output: >cd /data/fxa/LDAD/mesonet/qc >ls -l 20000113_1600 -rw-r--r-- 1 fxa fxalpha 1419768 Jan 13 16:53 20000113_1600 Looks about right (although file size is dependent on the number of LDAD mesonet obs). If the files are not there, then check that the MSAS internal files are there: >cd /awips/fxa/ldad/MSAS/qcms_data/mesonet >ls -l 20000113_1600 -rw-r--r-- 1 fxa fxalpha 1419768 Jan 13 16:53 20000113_1600 Note - issue the following command to check that proper links are set: >pwd /data/fxa/LDAD/qcms_data/mesonet *MSAS has its own directory conventions, used at all installations. Links are used on the WFO_Advanced system to point the "MSAS directories" to the "actual directories" (see create_MSAS_links.csh, in PCMS directory .../ldad/src/install_scripts/, on the running as2 system in directory /awips/fxa/ldad/bin). If the internal files aren't there, check the logs: >cd /awips/fxa/ldad/MSAS/logs >pwd /data/logs/fxa/ldad/MSAS/logs *"qcstg1_2log" and "qclatelog" are the logs to examine. They are produced by the WFOA_MSAS_QCstage1_2.run and WFOA_MSAS_QCstage1_2_late.run scripts, respectively, both of which are located at /awips/fxa/ldad/MSAS. * output files associated with qcstg1_2log and qclatelog are labeled "qcstg1_2-meso.out.YYJJJHH00_mm[_L]", where JJJ is the julian day (and mm is the minute of the run for WFOA_MSAS_QCstage1_2.run, and mm_L is the minute of the run for WFOA_MSAS_QCstage1_2_late.run). Note- /data/fxa/LDAD/mesonet/qc files can not be produced if /data/fxa/LDAD /mesonet/netCDF files are not being produced. 2. Check the final CSV output (ASCII files in CSV format are processed by the SHEF encoder so that mesonet data and QC results can be placed in the hydro database): >cd /awips/fxa/ldad/MSAS/qcms_data/csv >pwd /data/fxa/LDAD/qcms_data/csv >ls -l *.947779200 -rw-r--r-- 1 fxa fxalpha 950 Jan 13 17:06 947779200.06.msas_qc.SCHLNET -rw-r--r-- 1 fxa fxalpha 47500 Jan 13 16:18 947779200.18.msas_qc.RAWS -rw-r--r-- 1 fxa fxalpha 2470 Jan 13 16:23 947779200.23.msas_qc.SCHLNET -rw-r--r-- 1 fxa fxalpha 1140 Jan 13 16:33 947779200.33.msas_qc.SCHLNET -rw-r--r-- 1 fxa fxalpha 1140 Jan 13 16:43 947779200.43.msas_qc.SCHLNET -rw-r--r-- 1 fxa fxalpha 3420 Jan 13 16:48 947779200.48.msas_qc.RAWS -rw-r--r-- 1 fxa fxalpha 190 Jan 13 16:53 947779200.53.msas_qc.SCHLNET -rw-r--r-- 1 fxa fxalpha 38950 Jan 13 16:19 947779200.q3.msas_qc.RAWS -rw-r--r-- 1 fxa fxalpha 190 Jan 13 16:19 947779200.q3.msas_qc.SCHLNET Looks about right. The CSV files are named: utime.pd.msas_qc.pname where: utime = formatted Unix time version of the time (e.g., 947779200 for 20000113_1600) pd = minute of the QC run (e.g., "06", "11"), or "q3" msas_qc = the *.desc file to be used by LDAD processing, pname = data type from LDADinfo.txt of the provider (e.g., "SCHLNET") If the CSV files aren't there, check the qcstg1_2log and qclatelog files (see above). Note- /data/fxa/LDAD/qcms_data/csv files can not be produced if /data/fxa/LDAD/mesonet/netCDF files are not being produced. ---------------------------- Hourly Analysis and QC Cycle ----------------------------- The hourly cycle runs at 18 minutes after the hour. The first things to check are the final outputs. [Note- all examples are for 1/13/00 1600 GMT] 1. Make sure the netCDF grids produced by MSAS are OK: >cd /data/fxa/Grid/FSL/netCDF/MSAS >ls -l 20000113_1600 -rw-r--r-- 1 fxa fxalpha 245888 Jan 13 16:19 20000113_1600 *Looks about right. 2. Make sure the netCDF observation files produced by MSAS are OK: >cd /awips/fxa/ldad/MSAS/sfc_netcdf >pwd /data/fxa/LDAD/sfc_netcdf >ls -l 20000113_1600 -rw-r--r-- 1 fxa fxalpha 2565708 Jan 13 16:19 20000113_1600 *Looks about right. 3. Make sure QC messages produced by MSAS are OK: >cd /awips/fxa/ldad/MSAS/qc5 >pwd /data/fxa/LDAD/qcmessages/qc5 *Note, as in the subhourly cycle, links are used to point the "MSAS directories" to the "actual directories". >ls -l *16.dat -rw-r--r-- 1 fxa fxalpha 115813 Jan 13 16:19 qcobs16.dat -rw-r--r-- 1 fxa fxalpha 8886 Jan 13 16:19 qcsumry16.dat *There are actually 21 of these directories (qc0 - qc20). The one used in this example (qc5) is the biggest one, for ASOS stations. Others include: qc1 - for "SAO" (METAR manual), qc 2 - for "BUOY", qc3 - for "NPN", qc4 - for "AUTO" (METAR automated, but non-ASOS), and qc6 through qc20 for LDAD mesonet (first come, first serve). Directory qc0 is for overflow. The qcobsHH.dat and qcsumryHH.dat files get produced for each hour. If one qc directory works, they probably all do. 4. Make sure nothing crashed: First check the top-level MSAS directory: >cd /awips/fxa/ldad/MSAS > ls -l core -rw------- 1 fxa fxalpha 742996 Jan 13 16:19 core *Whoops! The core dump shouldn't have happened. (In this case, the textdb utility didn't work as an Informix library external to MSAS had not been properly re-installed following the system upgrade.) Now check for core files in all of the MSAS run directories. > ls -lR */core If the final output files aren't there, or if there is a core file, check the logs: >cd /awips/fxa/ldad/MSAS/logs >pwd /data/logs/fxa/ldad/MSAS/logs *"sfclog" is the main log to examine. It is produced by WFOA_MSAS_Surface.run in /awips/fxa/ldad/MSAS. If no indication of the error is found, check the other log files (asoslog, qcdaylog, qclatelog, qcstg1_2log, sfcnmclog). If the final outputs weren't there, check some of the MSAS internal files: Note that the most obvious problems will be that one or more of the directories will either not exist, or will be owned by ldad instead of fxa. Also, the following output files might differ hour by hour. If any individual files are missing, something's broken. If all files are missing, everything's broken. A. Check if output files are being produced by the individual programs in the hourly cycle: *(time is in YYJJJHH00 format) >cd /awips/fxa/ldad/MSAS/output >pwd /data/logs/fxa/ldad/MSAS/output >ls -l *.out.000131600 -rw-r--r-- 1 fxa fxalpha 574 Jan 13 16:19 accept.out.000131600 -rw-r--r-- 1 fxa fxalpha 4847 Jan 13 16:19 qcstg3.out.000131600 -rw-r--r-- 1 fxa fxalpha 902 Jan 13 16:19 reject.out.000131600 -rw-r--r-- 1 fxa fxalpha 4537 Jan 13 16:19 sfcanl.out.000131600 -rw-r--r-- 1 fxa fxalpha 208548 Jan 13 16:19 sfchqc.out.000131600 -rw-r--r-- 1 fxa fxalpha 252863 Jan 13 16:18 sfcing.out.000131600 -rw-r--r-- 1 fxa fxalpha 5787 Jan 13 16:19 sfcncdf.out.000131600 -rw-r--r-- 1 fxa fxalpha 35025 Jan 13 16:19 sfcver.out.000131600 *Looks normal. B. Check to see that observations are being ingest: >grep SRCN sfchqc.out.000131600 SRCN 0 ALL 0 SRCN 500 SAO 282 SRCN 520 BUOY 64 SRCN 205 NPN 14 SRCN 530 AUTO 363 SRCN 510 ASOS 869 SRCN 601 NWSRAWS 206 SRCN 602 SCHLNET 1 *Note that the local data sources (NWSRAWS & SCHLNET) being processed by MSAS are shown. New local sources are added as they are identified in the data being ingested from LDAD mesonet files. To start fresh (as with new installations) copy /awips/fxa/ldad/MSAS/fslparms/sfchqcin.initial to .../MSAS/fslparms/sfchqcin.dat. The .initial file contains national data sources only. C. Check to see that observations are being analyzed: >grep # sfcanl.out.000131600 -- # OF OBS FOR VARIABLE 1 = 1451 -- # OF OBS FOR VARIABLE 2 = 1412 -- # OF OBS FOR VARIABLE 3 = 1699 -- # OF OBS FOR VARIABLE 4 = 1699 -- # OF OBS FOR VARIABLE 5 = 1638 -- # OF OBS FOR VARIABLE 6 = 1690 -- # OF OBS FOR VARIABLE 7 = 1307 -- # OF OBS FOR VARIABLE 8 = 1063 -- # OF OBS FOR VARIABLE 9 = 1449 *Variables are in the following order: MSAS SLP, potential temperature, U, V, dew point, dewpoint depression, pressure change, NWS SLP, Altimeter. Numbers are for all data sources combined (SAO+BUOY+NPN+...), and represent observations that passed the QC checks. Analyses will not be generated if the number of METAR observations is less than 300. In that case, typical output of this grep command would read: -- # OF OBS FOR VARIABLE 8 = 133 -- NOT ENOUGH TO BE WRITTEN. Missing MSAS SLP analyses are usually the result of missing *.mig files (since MSLP is derived in MSAS as a function of Eta 700mb temp). See Eta Ingest Cycle. D. Check if any unusual errors have occurred: >cd /awips/fxa/ldad/MSAS/errors >pwd /data/logs/fxa/ldad/MSAS/errors >ls -l *.err.000131600 -rw-r--r-- 1 fxa fxalpha 0 Jan 13 16:19 qcstg3.err.000131600 -rw-r--r-- 1 fxa fxalpha 0 Jan 13 16:19 sfcanl.err.000131600 -rw-r--r-- 1 fxa fxalpha 0 Jan 13 16:19 sfchqc.err.000131600 -rw-r--r-- 1 fxa fxalpha 0 Jan 13 16:18 sfcing.err.000131600 -rw-r--r-- 1 fxa fxalpha 0 Jan 13 16:19 sfcncdf.err.000131600 -rw-r--r-- 1 fxa fxalpha 0 Jan 13 16:19 sfcver.err.000131600 *Zero indicates no errors. E. Check MSAS internal database files: >cd /awips/fxa/ldad/MSAS/maps_data >pwd /data/fxa/LDAD/maps_data >ls -l 000131600.* -rw-r--r-- 1 fxa fxalpha 18176 Jan 13 16:19 000131600.mby -rw-r--r-- 1 fxa fxalpha 225280 Jan 13 16:19 000131600.mfg -rw-r--r-- 1 fxa fxalpha 474240 Jan 13 16:19 000131600.msa -rw-r--r-- 1 fxa fxalpha 128000 Jan 13 16:18 000131600.mss -rw-r--r-- 1 fxa fxalpha 225280 Jan 13 16:19 000131600.msx -rw-r--r-- 1 fxa fxalpha 41 Jan 13 16:19 000131600.replist.error -rw-r--r-- 1 fxa fxalpha 46949 Jan 13 16:18 000131600.sri -rw-r--r-- 1 fxa fxalpha 200574 Jan 13 16:19 000131600.srq *The .mby is for buoy obs, the .msa for land-based sfc obs. If there's no data for one of those, there won't be a file, and that would be OK, if there's really no data coming in. If either of those files exists, the .mss, .msx, .sri, and .srq files should also exist. The .replist.error file should exist even if no others do, and there may or may not be a .replist file (in this example there isn't). ---------------- Eta Ingest Cycle ---------------- Eta grids are ingest by MSAS twice a day, at 05:37 & 17:37. 1. Check that the final output files are being produces: >cd /awips/fxa/ldad/MSAS/mdl_data >pwd /data/fxa/LDAD/mdl_data >ls -l total 9440 -rw-r--r-- 1 fxa fxalpha 962560 Jan 13 06:57 000130006.mig -rw-r--r-- 1 fxa fxalpha 225280 Jan 13 06:57 000130006.msi -rw-r--r-- 1 fxa fxalpha 962560 Jan 13 06:57 000130012.mig -rw-r--r-- 1 fxa fxalpha 225280 Jan 13 06:57 000130012.msi -rw-r--r-- 1 fxa fxalpha 962560 Jan 13 06:57 000130018.mig -rw-r--r-- 1 fxa fxalpha 225280 Jan 13 06:57 000130018.msi -rw-r--r-- 1 fxa fxalpha 962560 Jan 13 06:57 000130024.mig -rw-r--r-- 1 fxa fxalpha 225280 Jan 13 06:57 000130024.msi -rw-r--r-- 1 fxa fxalpha 962560 Jan 13 06:57 000130030.mig -rw-r--r-- 1 fxa fxalpha 225280 Jan 13 06:57 000130030.msi *This looks normal. *Note- Missing MSLP MSAS analyses are usually the result of missing *.mig files (since MSLP is derived in MSAS as a function of Eta 700mb temp). If the files, are not there, check the logs: >cd /awips/fxa/ldad/MSAS/logs >pwd /data/logs/fxa/ldad/MSAS/logs *sfcnmclog is the log to examine. It is produced by /awips/fxa/ldad/MSAS/ WFOA_MSAS_Sfcnmc.run. sfcnmc* are the appropriate "out" and "err" files. ---------------------- Daily QC Summary Cycle ---------------------- Program ASOS runs at 23:53 to process each days' data; program QCSTATS runs at 00:35 to process the previous days' data. To check that they are running properly: 1. Check for ASOS daily summaries: >cd /awips/fxa/ldad/MSAS/qc5 >pwd /data/fxa/LDAD/qcmessages/qc5 >ls -l *day* -rw-r--r-- 1 fxa fxalpha 8826 Jan 8 23:53 daysum0.dat -rw-r--r-- 1 fxa fxalpha 10297 Jan 9 23:54 daysum1.dat -rw-r--r-- 1 fxa fxalpha 11768 Jan 10 23:54 daysum2.dat -rw-r--r-- 1 fxa fxalpha 8826 Jan 11 23:54 daysum3.dat -rw-r--r-- 1 fxa fxalpha 8826 Jan 12 23:54 daysum4.dat -rw-r--r-- 1 fxa fxalpha 11768 Jan 6 23:53 daysum5.dat -rw-r--r-- 1 fxa fxalpha 10297 Jan 7 23:53 daysum6.dat -rw-r--r-- 1 fxa fxalpha 354084 Jan 8 23:53 qcday0.dat -rw-r--r-- 1 fxa fxalpha 353274 Jan 9 23:54 qcday1.dat -rw-r--r-- 1 fxa fxalpha 354489 Jan 10 23:54 qcday2.dat -rw-r--r-- 1 fxa fxalpha 354489 Jan 11 23:54 qcday3.dat -rw-r--r-- 1 fxa fxalpha 353274 Jan 12 23:54 qcday4.dat -rw-r--r-- 1 fxa fxalpha 353679 Jan 6 23:53 qcday5.dat -rw-r--r-- 1 fxa fxalpha 354084 Jan 7 23:53 qcday6.dat *In each of the 21 qc directories a daysum file and a qcday file is created once each day. (One weeks' worth of files are kept at all times.) Once a week there's also wksum & qcwk files, and once a month a mnthsum & qcmnth file are created. If the files aren't there, check the logs: >cd /awips/fxa/ldad/MSAS/logs >pwd /data/logs/fxa/ldad/MSAS/logs *asoslog is the log to check. It is produced by /awips/fxa/ldad/MSAS/ WFOA_MSAS_Asos.run. asos* are the appropriate "out" and "err" files. 2. Check for QCSTAT daily summaries: >cd /awips/fxa/ldad/MSAS/qcms_data/stats >pwd /data/fxa/LDAD/qcms_data/stats >ls -l *000120000 -rw-r--r-- 1 fxa fxalpha 8399 Jan 13 00:35 qcstats.000120000 *(time is in YYJJJHH00 format) If the files aren't there, check the logs: >cd /awips/fxa/ldad/MSAS/logs >pwd /data/logs/fxa/ldad/MSAS/logs *qcdaylog is the log to check. It is produced by /awips/fxa/ldad/MSAS/ WFOA_MSAS_QCday.run. qcstats* are the appropriate "out" and "err" files.