Case Archive (Radar Archive)

This is essentially just writing AWIPS data (currently, only radar data) permanently to archive media in such a way that it can be restored and displayed at a later time. However, there are various details about how the archiving needs to be done that have ended up making the case archiver system fairly complicated.

The user is able to specify a sub-set of data to be archived both in terms of a sub-set of products and in terms of a sub-time-range. And, to specify if just the data currently on the ingest disk is to be stored or if data is to be stored continuously as it comes in. The user can also specify a sub-set of products to restore of the products stored. There is also the fact that the case archiver uses the HP tape autoloader and must share it with the system backup task. It is critical that the case archiver not interfere with system backup.

Basic functional description:

Store:

When a user elects to store data via the archive manager the manager constructs and sends a store request to the caseArchiveServer running on one of the DS machines. When the server receives the request it invokes storeToCache.sh to copy the data from the AWIPS ingest disk to the archive cache disk. The data is then stored to archive media by a cron job, storeToTape.sh, which runs every 12 hours. This is true of continuous-stores as well as one-time-stores. With a continuous-store a store to cache will take place as soon as the user initiates the request and will also occur every time a store request is automatically sent to the server by contCacheUtil. ContCacheUtil runs every 90 minute to perform the intermediate stores when continuous store mode is on.

The cache disk directory name is the value of the configuration option dataCaseCache.dir. After each storeToTape the data on the cache disk is removed.

Each successful store request becomes a case archive "store session." These sessions are what the user selects during restore.

Restore:

When a user elects to restore data the archive manager constructs and sends a restore request to the caseArchiveServer. When the server receives the request it invokes restoreToCase.sh to restore the data from the archive media to the case directory on the restore disk. The restore disk directory name is the value of the configuration option dataCaseRestore.dir. The case directory will be created as a sub directory of the restore disk directory. The case directory name will be the name of the case with any spaces replaced by underscores.

Case Display:

When the user elects to display a case the archive manager invokes the displayCase script as a separate process that runs in the background on the Workstation. The manager then shuts down its parent D-2D and itself. The displayCase script continues to run. It waits for the parent D-2D to exit and then starts another D-2D with the environment variables and arguments set to run in case review mode with the selected case. When this D-2D exits, displayCase exits as well resulting in all the special settings being forgotten.

Case Archive store sessions v.s. OmniBack backup sessions:

A Case Archive store sessions is created each time a users request to store data is successful. Each of them results in data being copied to the archive cache disk, and to the same directory tree on the cache disk. If two store sessions overlap so that certain data files are in both, only one copy of the overlapping files will be on the disk and those files will belong to both store sessions.

When storeToTape.sh runs and all the data on the cache disk is stored to tape, then an OmniBack backup session is created. This backup session can include the data for many Case Archive store sessions. Detailed information about the backup sessions can be obtained via the OmniBack GUI.

Program Files:

Binary executables in ~fxa/bin

archiveWish	This is the tcl/tk interpreter for the Archive Manager. An archiveWish process is started whenever the user selects to run the Archive Manager. This process then interprets the archive tcl/tk scripts. Runs on the workstation nodes.
archiveTclsh	The tcl interpreter for the displayCase tcl script. Runs on the workstation nodes.
caseArchiveServer	The server executable that accepts media requests from the Archive Manager. Satisfies most of them by spawning shell scripts that then run OmniBack commands that access the tape or the OmniBack database tape info. Runs on the data server node.
contCacheUtil	Acts as an Archive Manager surrogate for sending the automatic store requests to the server for a continuous store. Is set to run every 90 minute as a cron job on the data server node.
rangeOfTimeStamps	Determines the oldest and newest awips data files in a set of directories. Is used to help determine the "oldest data time" for the radar data. Invoked by timeRangeOfData.csh.

Scripts in ~fxa/bin:

archive	The main tcl/tk script for the Archive Manager. This is the command that is run when the user elects to start the Archive Manager. It invokes archiveWish.
displayCase	Tcl script that re-starts the D-2D with the appropriate arguments and environment variable settings for it to come up in case archive mode. Invoked by the display case code in the Archive Manager.
omniGetFiles.sh	Gets a list of the files, and their sizes, on the cache disk or on the currently loaded tape that have valid times that fall within a given time range. Invoked by caseArchiveServer. Used to determine the amount of disk space needed for a restore.
omniGetLabel.sh	Gets the label of the currently loaded archive tape. Invoked by caseArchiveServer.
omniGetSessions.sh	Gets the number of OmniBack store sessions on a given tape. Invoked by caseArchiveServer and used to determine if a newly installed tape is empty and is therefore the new "current store tape."
omniVerifyMedia.sh	Verifies the label of the currently loaded tape using omnimm to read the tape. Used by caseArchiveServer and omniGetLabel.sh when complete verification is needed. Takes one to two minutes to run.
restoreToCase.sh	Performs the restore of data from the cache disk and/or the tape to a case directory. Invoked by caseArchiveServer.
storeToCache.sh	When the user elects to store data, this script is invoked by caseArchiveServer to copy the data to the cache disk.
storeToTape.sh	Stores the data on the cache disk to tape. Invoked every 12 hours by cron. This can be invoked manually to make sure any recent store sessions are actually on the current store tape when changing to a new current store tape. Make sure system backup is not running or about to run, and run the script as fxa on a DS.
tarCopy.sh	Uses tar to copy files in the current working directory to the destination directory. The copies will have the same path relative to the destination directory that they had with the current directory. Invoked from storeToCache.sh to copy the files.
timeRangeOfData.csh	Determines the oldest and newest awips data files in the radar data. Is used to determine the "oldest data time" for the radar data. Invoked by caseArchiveServer.

Tcl/tk scripts in ~fxa/lib/tcl/caseArchive:

The rest of the archive manager tcl/tk scripts. Invoked from "archive."

Data files in ~fxa/data/caseArchive:

The archive manager data files. Mostly the gif files for the archive manager icons.

Configuration Options:

These are found in ~fxa/data/localization/nationalData/ws.config.

dataCaseRestore.dir	Its value is the path name of the directory where data cases are restored. It should be a dedicated disk partition of 600 MBytes or more. It will almost certainly be set to /data/archive_restore on the ds.
dataCaseCache.dir	Its value is the path name of the directory where stored data is cached before it is transferred to tape. It should be a dedicated disk partition of 400 MBytes or more. Will probably be /data/archive_cache.
archiveDBDir.dir	Its value is the path name of the directory where the case archive database will reside. By default it will take up part of the dataCaseRestore.dir partition. Will probably be /data/archive_restore/db.
dataCaseCacheReportLimit.percent	Its value is the percentage of cache disk space that must be used before warnings messages will appear on the D2D status bar. If the warnings do appear then the cache disk is close to being full and archive store requests may fail.
caseCommonData.dir	Its value is a space separated list of all the directories under $FXA_DATA which each restored case must have access to in order for the case to display. A case directory must contain symbolic links to each of these directories.
archiveDriveAccessLimits.timeStr	Its value is the time range during which the case archive software is allowed to access to the tape autoloader. The time outside this range is reserved for system backup.

Database Files:

All of these files and directories are under the archive data base directory which is specified by the archiveDBDir.dir config option.

.lockArchiveCache	This is the locking file used by both storeToCache.sh and storeToTape.sh to keep them from accessing the archive cache directory at the same time. It will only exist while one of these two scripts is running.
continuousStoreOn	Exist as a zero length flag file if continuous store is on. Does not exist if continuous store is off.
currentStoreMedium	Contains the label of the current store tape. Its contents will only change when a newly initialized tape is loaded in slot 2 of the autoloader.
restoreDirLists.*	Temporary files containing the list of directories to restore. One will exist only as long as restoreToCase.sh is running.
startTimeStamp.* & endTimeStamp.*	Temporary zero length files who's modification times reflect the start and end date/times of a store. A pair of these will exist only as long as storeToCache.sh is running.
dirListsFile.*	Temporary files each containing the list of directories to store. One will exist only as long as storeToCache.sh is running.
getFiles.newer.* & getFiles.older.*	Temporary zero length files who's modification times reflect the start and end date/times of a restore respectively. A pair of these will exist only as long as restoreToCase.sh is running.
emptyDir	An empty directory that omniGetFiles.sh cd's to. Keeps ll with no arguments from producing any output.
omniTapesList	Used by omniGetLabel.sh to keep track of multiple tape labels when more than one archive tape is in the OmniBack database. This occurs when restoring from an old tape. Will be deleted once the restore is completed and the OmniBack database once again only knows about the current store tape.
scriptLogs/restoreFind.out	Used by restoreToCase.sh to contain any directories selected for restore that were actually found on the cache disk. Allows the script to determine if files were successfully restored from the cache disk. Also used by storeToCache.sh in much the same way - to determine if any files actually existed for the product set selected and the time range selected and were stored.

The following files and directories are under the clientDB sub directory of the archive data base directory. This directory contains the files and directories for the archive manager's that run on the workstations. The word client refers to the fact that all the archive manager's are clients of the caseArchiveServer.

.lockfile The client database lock file. Insures that only one archive manager can access the client database at a time.

contStoreMS The media session database file for continuous store. Basically contains the information on the most recently submitted product set.

archiveMedia The directory that contains the database files for each archive media. Each file under this directory has the same name as the associated media.

dataCases The directory that contains the database files for each restored case.

Log Files:

Files under the archive data base directory which is specified by the archiveDBDir.dir config option.

scriptLogs/*.sh.out	Output "debug" files for the major archive shell scripts invoked by caseArchiveServer. Only one exists for each script and gets replaced the next time the script runs.
/data/logs/fxa/<date>/ caseArchiveServer<PID><node_name><time>:	The AWIPS log file for the caseArchiveServer on the ds node.
/data/logs/fxa/display/<XdisplayID>/<date>/ fxaWish<PID>.children/archiveWish<PID>:	The AWIPS log file for the archive manager on a ws node.

OmniBack Configuration:

Configuring the OmniBack software for case archive is done during the AWIPS installation. Here are the basics:

Archive user:	This is the fxa user, since the caseArchiveServer runs as fxa and all the OmniBack commands are issued by the server. Fxa should be set up as an OmniBack Admin user.
Archive logical device:	"Archive_ldev" which should be configured to use slot 2 of the autoloader.
Archive data list:	"Archive_datalist" which should be set to back up everything in the /data/archive_cache filesystem on the ds machines. Note that if the archive cache partition directory needs to change then it needs to be changed here as well as in the archiveDBDir.dir config option.
Archive Media pool:	"Archive_pool"
Media label:	It must start with Archive_pool. Example: Archive_pool_1

Archive Tape manipulation instructions:

Slot 2 of both autoloader magazines is reserved for the case archive tape. The currently installed magazine will normally contain the current store tape. Both store and restore functionality is enabled while the current store tape is in this slot. It can be temporarily replaced with an old tape for restore only. When the current store tape is full it will become an old tape when it is replaced with a newly initialized tape which will then become the new current store tape.

Care must be taken not to change the archive tape when system backup tasks are running or are about to run.

Initializing the first archive current store tape:

There can only be one case archive "current store tape" at a time. This means that when initializing the magazines, magazine 1 should contain an archive tape in slot 2 and that tape should be initialized with the other tapes. On switching to magazine 2 the archive tape must be moved from magazine 1 to magazine 2. The other tapes in magazine 2 must then be initialized but not the archive tape. When magazines are switched on the standard weekly basis the archive tape must be removed from the magazine being taken out and put into the magazine being put in. This means that the magazine not currently in the autoloader will have an empty slot 2. When the tape is 80% full announcements will start being seen on the D-2D status bar informing users that the tape needs to be changed soon. That should provide plenty of time to allow the ESA to put in and initialize a new "store" tape. As soon as a newly initialized tape appears in the drive the software will accept it as the new store tape.

To Replace the current store tape:

When the current store tape is 80% full, messages to that effect will start appearing on the D-2D's general status bar. These messages come from the storeToTape.sh script which runs as a cron job every 12 hours, so one will appear every 12 hours on every workstation. The tape does not have to be replaced immediately if, in your estimate, it will still be quite a while before it fills up. Since the percentage of space used on the tape is reported in these messages, as more case archiving is done they will tell you just how full the tape is. When the tape is 90% full the messages are assigned an urgent status and will be displayed in a red banner that the user has to acknowledge. At this point the tape must be replaced very soon. Make sure the cache disk is empty and that no store requests are in progress. This may require waiting until storeToTape.sh runs or running it manually.

Bring up the OmniBack GUI and select the Media button to bring up the "Media Pools" dialog. Double click on the Archive_pool entry to bring up the Archive_pool "Media Management" dialog.
Recycle the current store medium: Use the View pull down menu to select "All Media". Select the current archive medium - it should be the only media in the list. Use the Edit pull down menu to select "Recycle". The media is now no longer protected against overwrites so make sure nothing is stored to it before you export it.
Export the medium: Select the current archive medium. Use the Edit pull down menu to select "Export."
Physically remove the tape from slot 2 of currently loaded autoloader magazine and physically write protect it. It is now an old tape that cannot be stored to anymore but data can be restored from it.
Load a fresh archive tape: Select a fresh tape and physically label it with a tape name that is different from any other archive tape. A tape label sequence such as Archive_pool_1, Archive_pool_2, etc. is recommended for each successive store tapes. Put the tape into slot 2 of the current autoloader magazine and put the magazine back in the drive.
Initialize the new medium: Use the View pull down menu to select "Magazines." Use the Edit pull down menu to select "Init Magazine." The "Initialize Magazine" dialog will display. Fill in the "Logical Device" field by selecting the ... button on the right. Double click on the "Archive_ldev" entry in the resulting list dialog. Elect to specify the tape capacity and set the capacity to 4000 MB for a DDS-2 tape or 12000 MB for a DDS-3 tape. Turn on the "Force Initialization:" toggle button. Select the OK button to start the initialization.
Change the OmniBack medium label to match the tape's physical label: OmniBack will assign a tape label that is unlikely to match the name you put on the tape. Change it by using the View pull down menu to select "All Media". Select the current archive medium that you just initialized. Use the Edit pull down menu to select "Modify." A "Modify Medium" dialog will appear that will allow you to set the tape label.

Switch to an old tape for restore:

When a user selects to restore from an older tape in the restore tab of the archive manager they will be informed that the old tape needs to be physically installed in the autoloader. Follow these steps to temporarily replace the current store tape with the old tape.

Take the current store tape out of slot 2 of the currently loaded magazine and put the old tape in.
Bring up the OmniBack GUI and select the Media button to bring up the "Media Pools" dialog. Double click on the Archive_pool entry to bring up the Archive_pool "Media Management" dialog.
Import the old tape: Use the View pull down menu to select "Magazines." Use the Edit pull down menu to select "Import." The "Import" dialog will display. Fill in the "Logical Device" field by selecting the ... button on the right. Double click on the "Archive_ldev" entry in the resulting list dialog. Select the OK button to start the import. This will take a while as all the OmniBack store sessions must be read off the tape and added to the OmniBack database. Once this is done there will be two entries in the Archive_pool "Media Management" dialog's media list - one for the current store tape and one for the old tape you just loaded. It is very important that you leave the current store tape entry alone during this entire procedure.
Fix the old tape's label: Use the View pull down menu to select "All Media". Select the entry for the old tape. Use the Edit pull down menu to select "Modify." A "Modify Medium" dialog will appear that will allow you to set the tape label. Set it to be the same as the tape's physical label.
Inform the user that the old tape is ready for restore. The user may have already been informed by the Archive Manager GUI. Remind the user to let you know when they are done.
Export the old tape: When the user is done restoring from the old tape use the View pull down menu to select "All Media". Select the old tape's entry in the media list. Use the Edit pull down menu to select "Recycle". Select the old tape's entry again and then use the Edit pull down menu to select "Export." Be very sure that its the old tape's list entry that you select for these actions.
Physically remove the old tape from the drive and put the current store tape back in.
Scan the current store tape: Use the View pull down menu to select "Magazines". Use the Edit pull down menu to select "Scan". The "Scan" dialog will display. Fill in the "Logical Device" field by selecting the ... button on the right. Double click on the "Archive_ldev" entry in the resulting list dialog. Select the OK button to start the scan. This will take a minute or two.

The Archive Manager's restore tab should now display the current store tape as the currently loaded tape.

Removing all of a Tape's Sessions from the Restore Dialog:

Doing this is useful when archive manager startup is taking too long, or when the contCacheUtil process starts taking up way too much cpu on the ds. The latter condition should only occur in AWIPS version prior to 4.3.3.

For 4.3.3 the only way to alleviate slow archive manager start up is to initialize a new current store tape - see "To Replace the current store tape" above. The current store tape does not need to be full before it is replaced. Of course this will waste tape space.

For 4.2.x removing the sessions for an old tape will alleviate both conditions above. It requires moving the tape's session files in the case archive database to another location. It can be done for all of the old tapes if needed, and if that is not enough the current store tape can be replaced and then this procedure can be done for the "new" old tape as well.

Make sure the archive manager is not running anywhere.
Log on to a terminal window as fxa:

cd /data/archive_restore/db/clientDB/archiveMedia

mv <old_tape_name> ..

Start the archive manager and start doing stores if desired.

.lockfile	The client database lock file. Insures that only one archive manager can access the client database at a time.
contStoreMS	The media session database file for continuous store. Basically contains the information on the most recently submitted product set.
archiveMedia	The directory that contains the database files for each archive media. Each file under this directory has the same name as the associated media.
dataCases	The directory that contains the database files for each restored case.