This is essentially just writing AWIPS data (currently, only radar data) permanently to archive media in such a way that it can be restored and displayed at a later time. However, there are various details about how the archiving needs to be done that have ended up making the case archiver system fairly complicated.
The user is able to specify a sub-set of data to be archived both in terms of a sub-set of products and in terms of a sub-time-range. And, to specify if just the data currently on the ingest disk is to be stored or if data is to be stored continuously as it comes in. The user can also specify a sub-set of products to restore of the products stored. There is also the fact that the case archiver uses the HP tape autoloader and must share it with the system backup task. It is critical that the case archiver not interfere with system backup.
When a user elects to store data via the archive manager the manager constructs and sends a store request to the caseArchiveServer running on one of the DS machines. When the server receives the request it invokes storeToCache.sh to copy the data from the AWIPS ingest disk to the archive cache disk. The data is then stored to archive media by a cron job, storeToTape.sh, which runs every 12 hours. This is true of continuous-stores as well as one-time-stores. With a continuous-store a store to cache will take place as soon as the user initiates the request and will also occur every time a store request is automatically sent to the server by contCacheUtil. ContCacheUtil runs every 90 minute to perform the intermediate stores when continuous store mode is on.
The cache disk directory name is the value of the configuration option dataCaseCache.dir. After each storeToTape the data on the cache disk is removed.
Each successful store request becomes a case archive "store session." These sessions are what the user selects during restore.
When a user elects to restore data the archive manager constructs and sends a restore request to the caseArchiveServer. When the server receives the request it invokes restoreToCase.sh to restore the data from the archive media to the case directory on the restore disk. The restore disk directory name is the value of the configuration option dataCaseRestore.dir. The case directory will be created as a sub-directory of the restore disk directory. The case directory name will be the name of the case with any spaces replaced by underscores.
When the user elects to display a case the archive manager invokes the displayCase script as a separate process that runs in the background on the Workstation. The manager then shuts down its parent D-2D and itself. The displayCase script continues to run. It waits for the parent D-2D to exit and then starts another D-2D with the environment variables and arguments set to run in case review mode with the selected case. When this D-2D exits, displayCase exits as well resulting in all the special settings being forgotten.
A Case Archive store sessions is created each time a users request to store data is successful. Each of them results in data being copied to the archive cache disk, and to the same directory tree on the cache disk. If two store sessions overlap so that certain data files are in both, only one copy of the overlapping files will be on the disk and those files will belong to both store sessions.
When storeToTape.sh runs and all the data on the cache disk is stored to tape, then an OmniBack backup session is created. This backup session can include the data for many Case Archive store sessions. Detailed information about the backup sessions can be obtained via the OmniBack GUI.
| archiveWish | This is the tcl/tk interpreter for the Archive Manager. An archiveWish process is started whenever the user selects to run the Archive Manager. This process then interprets the archive tcl/tk scripts. Runs on the workstation nodes. |
| archiveTclsh | The tcl interpreter for the displayCase tcl script. Runs on the workstation nodes. |
| caseArchiveServer | The server executable that accepts media requests from the Archive Manager. Satisfies most of them by spawning shell scripts that then run OmniBack commands that access the tape or the OmniBack database tape info. Runs on the data server node. |
| contCacheUtil | Acts as an Archive Manager surrogate for sending the automatic store requests to the server for a continuous store. Is set to run every 90 minute as a cron job on the data server node. |
| rangeOfTimeStamps | Determines the oldest and newest awips data files in a set of directories. Is used to help determine the "oldest data time" for the radar data. Invoked by timeRangeOfData.csh. |
| archive | The main tcl/tk script for the Archive Manager. This is the command that is run when the user elects to start the Archive Manager. It invokes archiveWish. |
| displayCase | Tcl script that re-starts the D-2D with the appropriate arguments and environment variable settings for it to come up in case archive mode. Invoked by the display case code in the Archive Manager. |
| omniGetFiles.sh | Gets a list of the files, and their sizes, on the cache disk or on the currently loaded tape that have valid times that fall within a given time range. Invoked by caseArchiveServer. Used to determine the amount of disk space needed for a restore. |
| omniGetLabel.sh | Gets the label of the currently loaded archive tape. Invoked by caseArchiveServer. |
| omniGetSessions.sh | Gets the number of OmniBack store sessions on a given tape. Invoked by caseArchiveServer and used to determine if a newly installed tape is empty and is therefore the new "current store tape." |
| omniVerifyMedia.sh | Verifies the label of the currently loaded tape using omnimm to read the tape. Used by caseArchiveServer and omniGetLabel.sh when complete verification is needed. Takes one to two minutes to run. |
| restoreToCase.sh | Performs the restore of data from the cache disk and/or the tape to a case directory. Invoked by caseArchiveServer. |
| storeToCache.sh | When the user elects to store data, this script is invoked by caseArchiveServer to copy the data to the cache disk. |
| storeToTape.sh | Stores the data on the cache disk to tape. Invoked every 12 hours by cron. This can be invoked manually to make sure any recent store sessions are actually on the current store tape when changing to a new current store tape. Make sure system backup is not running or about to run, and run the script as fxa on a DS. |
| tarCopy.sh | Uses tar to copy files in the current working directory to the destination directory. The copies will have the same path relative to the destination directory that they had with the current directory. Invoked from storeToCache.sh to copy the files. |
| timeRangeOfData.csh | Determines the oldest and newest awips data files in the radar data. Is used to determine the "oldest data time" for the radar data. Invoked by caseArchiveServer. |
The rest of the archive manager tcl/tk scripts. Invoked from "archive."
The archive manager data files. Mostly the gif files for the archive manager icons.
These are found in ~fxa/data/localization/nationalData/ws.config.
| dataCaseRestore.dir | Its value is the path name of the directory where data cases are restored. It should be a dedicated disk partition of 600 MBytes or more. It will almost certainly be set to /data/archive_restore on the ds. |
| dataCaseCache.dir | Its value is the path name of the directory where stored data is cached before it is transfered to tape. It should be a dedicated disk partition of 400 MBytes or more. Will probably be /data/archive_cache. |
| archiveDBDir.dir | Its value is the path name of the directory where the case archive database will reside. By default it will take up part of the dataCaseRestore.dir partition. Will probably be /data/archive_restore/db. |
| dataCaseCacheReportLimit.percent | Its value is the percentage of cache disk space that must be used before warnings messages will appear on the D2D status bar. If the warnings do appear then the cache disk is close to being full and archive store requests may fail. |
| caseCommonData.dir | Its value is a space separated list of all the directories under $FXA_DATA which each restored case must have access to in order for the case to display. A case directory must contain symbolic links to each of these directories. |
| archiveDriveAccessLimits.timeStr | Its value is the time range during which the case archive software is allowed to access to the tape autoloader. The time outside this range is reserved for system backup. |
All of these files and directories are under the archive data base directory which is specified by the archiveDBDir.dir config option.
| .lockArchiveCache | This is the locking file used by both storeToCache.sh and storeToTape.sh to keep them from accessing the archive cache directory at the same time. It will only exist while one of these two scripts is running. |
| continuousStoreOn | Exist as a zero length flag file if continuous store is on. Does not exist if continuous store is off. |
| currentStoreMedium | Contains the label of the current store tape. Its contents will only change when a newly initialized tape is loaded in slot 2 of the autoloader. |
| restoreDirLists.* | Temporary files containing the list of directories to restore. One will exist only as long as restoreToCase.sh is running. |
| startTimeStamp.* & endTimeStamp.* | Temporary zero length files who's modification times reflect the start and end date/times of a store. A pair of these will exist only as long as storeToCache.sh is running. |
| dirListsFile.* | Temporary files each containing the list of directories to store. One will exist only as long as storeToCache.sh is running. |
| getFiles.newer.* & getFiles.older.* | Temporary zero length files who's modification times reflect the start and end date/times of a restore respectively. A pair of these will exist only as long as restoreToCase.sh is running. |
| emptyDir | An empty directory that omniGetFiles.sh cd's to. Keeps ll with no arguments from producing any output. |
| omniTapesList | Used by omniGetLabel.sh to keep track of multiple tape labels when more than one archive tape is in the OmniBack database. This occurs when restoring from an old tape. Will be deleted once the restore is completed and the OmniBack database once again only knows about the current store tape. |
| scriptLogs/restoreFind.out | Used by restoreToCase.sh to contain any directories selected for restore that were actually found on the cache disk. Allows the script to determine if files were successfully restored from the cache disk. Also used by storeToCache.sh in much the same way - to determine if any files actually existed for the product set selected and the time range selected and were stored. |
The following files and directories are under the clientDB sub-directory of the archive data base directory. This directory contains the files and directories for the archive manager's that run on the workstations. The word client refers to the fact that all the archive manager's are clients of the caseArchiveServer.
| .lockfile | The client database lock file. Insures that only one archive manager can access the client database at a time. |
| contStoreMS | The media session database file for continuous store. Basically contains the information on the most recently submitted product set. |
| archiveMedia | The directory that contains the database files for each archive media. Each file under this directory has the same name as the associated media. |
| dataCases | The directory that contains the database files for each restored case. |
Files under the archive data base directory which is specified by the archiveDBDir.dir config option.
| scriptLogs/*.sh.out | Output "debug" files for the major archive shell scripts invoked by caseArchiveServer. Only one exists for each script and gets replaced the next time the script runs. |
| /data/logs/fxa/<date>/ caseArchiveServer<PID><node_name><time>: |
The AWIPS log file for the caseArchiveServer on the ds node. |
| /data/logs/fxa/display/<XdisplayID>/<date>/ fxaWish<PID>.children/archiveWish<PID>: |
The AWIPS log file for the archive manager on a ws node. |
Configuring the OmniBack software for case archive is done during the AWIPS installation. Here are the basics:
| Archive user: | This is the fxa user, since the caseArchiveServer runs as fxa and all the OmniBack commands are issued by the server. Fxa should be set up as an OmniBack Admin user. |
| Archive logical device: | "Archive_ldev" which should be configured to use slot 2 of the autoloader. |
| Archive data list: | "Archive_datalist" which should be set back up everything in the /data/archive_cache filesystem on the ds machines. Note that if the archive cache partition directory needs to change then it needs to be changed here as well as in the archiveDBDir.dir config option. |
| Archive Media pool: | "Archive_pool" |
Slot 2 of both autoloader magazines is reserved for the case archive tape. The currently installed magazine will normally contain the current store tape. Both store and restore functionality is enabled while the current store tape is in this slot. It can be temporarily replaced with an old tape for restore only. When the current store tape is full it will become an old tape when it is replaced with a newly initialized tape which will then become the new current store tape.
Care must be taken not to change the archive tape when system backup tasks are running or are about to run.
There can only be one case archive "current store tape" at a time. This means that when initializing the magazines, magazine 1 should contain an archive tape in slot 2 and that tape should be initialized with the other tapes. On switching to magazine 2 the archive tape must be moved from magazine 1 to magazine 2. The other tapes in magazine 2 must then be initialized but not the archive tape. When magazines are switched on the standard weekly basis the archive tape must be removed from the magazine being taken out and put into the magazine being put in. This means that the magazine not currently in the autoloader will have an empty slot 2. When the tape is 80% full announcements will start being seen on the D-2D status bar informing users that the tape needs to be changed soon. That should provide plenty of time to allow the ESA to put in and initialize a new "store" tape. As soon as a newly initialized tape appears in the drive the software will accept it as the new store tape.
When the current store tape is 80% full, messages to that effect will start appearing on the D-2D's general status bar. These messages come from the storeToTape.sh script which runs as a cron job every 12 hours, so one will appear every 12 hours on every workstation. The tape does not have to be replaced immediately if, in your estimate, it will still be quite a while before it fills up. Since the percentage of space used on the tape is reported in these messages, as more case archiving is done they will tell you just how full the tape is. When the tape is 90% full the messages are assigned an urgent status and will be displayed in a red banner that the user has to acknowledge. At this point the tape must be replaced very soon. Make sure the cache disk is empty and that no store requests are in progress. This may require waiting until storeToTape.sh runs or running it manually.
When a user selects to restore from an older tape in the restore tab of the archive manager they will be informed that the old tape needs to be physically installed in the autoloader. Follow these steps to temporarily replace the current store tape with the old tape.
The Archive Manager's restore tab should now display the current store tape as the currently loaded tape.