Dual Polarization work for OB9.

Updated Mar 12 2008

http://www-sdd.fsl.noaa.gov/~ramer/noaa/dualPolOB9/dualPolOB9.html

The basic capability for dealing with dual polarization radar data was first introduced into AWIPS for the OB8.3 release. However, the OB8.3 Dual Polarization work was mostly done just to support training; very few sites will actually have access to this data in real time while they are using the OB8.3 release. The OB9 release is the first release that will be supporting significant operational use of dual polarization radar data. As such, there is still substantial work being done in OB9 with regards to dual polarization radar data.

Here is a functional breakdown of this work; each item will be discussed in detail in the sections that follow:

We will conclude with a discussion of some issues that still may need to be resolved.


Reorganized Dual Polarization focused radar menus.

Most of the work designing this new radar menu layout has actually been done by people at the Warning Decision Training Branch. GSD has been involved in evaluating the new layout and integrating it into OB9. The new layout will not be in place by default when OB9 is installed, because most sites will not have dual polarization radar data available at that time. A turnkey script has been created, which can be found at the following path:

Running this script will convert the radar menu layout to the new dual polarization focused layout. There is an option to back this out as well. This turnkey script will eventually have to also do something with the default RPS lists, but that has not been engineered yet.


Panel/Combo Rotate.

With the introduction of dual polarization radar data, there are now eight major types of imagery available for each scan in a volume; reflectivity, velocity, storm relative velocity, spectrum width, differential reflectivity, correlation coefficient, specific differential phase, and hydrometeor classification. The one means we have in D-2D for loading eight images into the D-2D at once is by using the four panel mode. However, the four panel mode has two disadvantages. First, the individual panes in the four panel are lower resolution than viewing a single image covering the whole pane. Second, it is sometimes hard to visually compare multiple images from different panels and know for sure when you are looking at data for the same geographic location.

The Panel/Combo Rotate functionality is an attempt to mitigate some of these issues. In OB9, if one has a four panel product loaded, hitting the Delete key (in the set of six keys above the arrows and to the left of the numeric keypad) will enter the Panel/Combo Rotate mode. In this mode one of the panels will occupy the whole pane, but all the data will still be in the memory of the D-2D and quickly available for viewing. When the Delete key is first selected, one will be viewing the left combo of the upper left panel, as shown:

Notice the small blue cross in the lower left of the display; that will always be there if one is in Panel/Combo Rotate mode. Continuing to select the Delete key will first rotate though any available left combos in the remaining panels, and then through any available right combos in the panels. There are up to eight possible configurations before returning to the initial state. Selecting the Backspace key will back up, and selecting the End key will leave Panel/Combo Rotate mode. Furthermore, the numbers 1 through 4 on the regular keyboard will select the left combo for panels 1 through 4 and numbers 5 through 8 will select the right combo for panels 1 through 4. There are also third button popup options over the display area for entering the mode and rotating through the states of the mode.

One should note that the Panel/Combo Rotate functionality is not specific to radar; it works for any data set loaded as a four panel.


Sampling of non-viewable panels.

Normally, when one is in Panel/Combo Rotate mode, sampling will only show data for those images that are actually being viewed. Using the third button popup over the display area, one can activate a mode where data from all panels is sampleable even though one is only viewing one panel. Below are screen shots that show the activation mechanism and an example of the output from such a sample operation:

Note that the format of the sample output for each component product is modified for this prototype. Because of the large volume of output this can produce, the distance and bearing to the radar and height information has been removed, and to avoid ambiguity as to which output is for which tilt, the tilt angle has been added. The exact details as to how to modify the sampling output when this feature is invoked will undoubtedly go through some itterations.

There is an issue with this feature that for now can only be partially dealt with. If one loads a four panel with a large number of frames and then immediately enters Panel/Combo Rotate mode, there will be many frames for which the non-viewable panels never get rendered. If one steps to one of these frames and invokes the All Panel Sample, the panels that were never rendered will not contribute to the sample output because those depictables will not yet have data. The partial solution is to add a new virtual method in the Depictable class called verifyDataForSample. It is called before a depictable is used for sampling, and as the name suggests its job is to pull in the data to support sampling if it does not yet exist within the depictable. The reason this is only a partial solution is that the concrete implementation for this method has not been coded for every depictable. So far, it has been coded for Radar, Satellite, and plan view plots, and GSD will soon code it for all VB access. However, that is as far as this will go for now. If there are other GSD written depictables for which this is an issue, we are going to rely on others to identify them, and then GSD can implement the verifyDataForSample method. For depictables that were not written by GSD, it will be left to the originating devorg to implement that method; of course GSD can answer questions for them.

Furthermore, we are going to rely on others to identify those cases where the text of the sampling needs to be different for the All Panel Sample case. If such cases are identified, GSD will expect a concrete recommendation for what the text should look like. As the previous screen shot shows, we have already tweaked the format for All Panel Sample for radial radar products.


Feature following zoom.

When one is zoomed in over a very small area to be able to view a feature in detail, it is often problematic to see that level of detail over time. This is because animation will often cause the feature to move into and then leave the field of view. In OB9, there is a mechanism that will allow one to follow a feature of interest even though one is zoomed in to a small area.

First, one needs to identify the location and motion of the feature of interest. This is done by loading the Distance Speed tool and placing the tracking icon over the feature of interest in multiple frames. The tracker in warnGen will work for this as well. Once one is satisfied that the tracking icon is following the feature of interest, the feature following zoom functionality is activated by loading the tool of the same name off the Tools menu:

Once the overlay labeled Feature Following Zoom is loaded, the center of the zoom area will track with the Distance Speed icon. Toggling that overlay off will resume the standard zooming behavior, and toggling it back on will reinvoke the feature following zoom.


Ensure all workstation features apply to volume browser radar display.

Previous to OB9, there were several core workstation features that did not work in the case where radar data was loaded into the volume browser. First, bundles did not remember which radar was used when radar data was loaded from the volume browser. Now, if a bundle is loaded with Original selected on the procedure menu, it will use the same radar as was used when the bundle was originally created, regardless of the current state of the "Home" selection. If Current is selected, then the current "Home" radar will be used. Another feaure that now works with procedures is that one can change which radar is used for a volume browser load via the Alter Bundle mechanism. Finally, the Data Scale feaure now does something meaningful for the case of loading radar from the volume browser.


Additional CAPPI levels for volume browser radar display.

In order to better leverage the advantages of dual polarization data, it was decided to add many levels to the available set for displaying radar data through the volume browser. In OB8.3, some height levels were added, and in OB9 we are adding several temperature levels.

This example shows reflectivity being displayed on the -5C level, but all the dual polarization fields can be displayed in this manner as well. There were discussions about also adding some height layers in OB9, and these details can be still be worked out.


Enhanced procedure patching.

One of the reasons we waited until OB9 to add all the CAPPI levels desired was that some of the desired levels were already being implemented in common field customizations. When levels are moved from being customized to being part of the default set, the indexing will change and as such procedures created using those levels will cease to function. This is because changing the indexing changes the depict keys made for products based on those levels, and procedures are at their core just a list of depict keys to load.

In OB9, to avoid causing problems by making these levels part of the default system, we are patching procedures. In the past, when changes had caused major remapping of the keyspace, we had made procedure patching part of the install. We did this in 5.2 (not OB5.2, but pre OB1 5.2) and in OB3. For the most part procedure patching in these releases was successful. We also did a very minor procedure patch in OB6, to change bundles made with the old AVN211 source to be bundles for the GFS212.

In OB9 we are going a step further in patching procedures, which should mean that we never have to do this again. For each depict key in the bundle, we are adding a corresponding string key. For volume browser loads, the key itself can be parsed into a source index, a plane index, a field index, and a display type index. There are tables for sources, planes, and fields, which connect an index to a particular item for each of these. Each of these items has an ASCII id defined in the table, and the string key for volume browser loads is in essence a catenation of the ids for source, plane, field, and display type. The procedure patching strategy is: use the pre-OB9 localization environment to compute string keys for each depict key, and then use the OB9 localization environment to convert those string keys back into depict keys, but the string keys will remain in the bundle. Furthermore, the way loading bundles works has changed. The code first looks at the string key and if it exists and can be converted into a useable depict key, then that key will be loaded. Otherwise, the depict key in the bundle will be loaded. This means that in the future if the indexing again changes, but the meaning of the ASCII ids does not, then no matter how much the index space gets scrambled, the procedures should continue to work transparently. Radar depict keys also have a well defined set of string keys in OB9. The generic fragment is entered into radarDepictKeys.template, tdwrDepictKeys.template, and asrArsrDepictKeys.template as a comma delimited item immediately after the depict key. For radar specific keys, the radar id is prepended to the generic fragment. For now, no other data sets have been assigned string keys, but it is just busy work to do so. For those products that do not get assigned string keys, procedures will continue to function operating solely based on the depict keys. Of course, with those products vigilance is still required so that the relationship between depict keys and products to load does not change.


Issues.

Removal of 3 bit products from the data keys for Best Res displays.

There have been occasional reports of poor performance when loading Best Res All Tilt displays. One thing we could do to mitigate this is to remove 3 bit products from the data keys for Best Res displays. None of the 3 bit products are directly selectable on the menus anymore; one can only display them if a higher bit depth selector is used and there are no higher bit depth products available for the same scan and tilt.

For velocity, there are three flavors of 3 bit product. a 0.25km that goes out to 60km, a 0.5 km that goes out to 115km, and a 1km that goes out to 230km. It would seem that at this point one would be sufficient.

For reflectivity there are also three flavors, a 1km that goes out to 230km, a 2km that goes out to 460km, and a 4km that goes out to 460km. Again, it would seem one would be sufficient.

Right now, the default depict keys for base reflectivity and velocity have eight data keys associated with each of them, and three of those are for 3 bit data. More data keys means inventories take incrementally longer. As a first step, perhaps it would not be unreasonable to take all the 3 bit data keys out of the Best Res products for reflectivity and velocity; it would still be possible to access them through the 4bit menus.

For that matter, it seems like we could do the same thing with the 4km 4 bit reflectivity. That is, make it only available from the 4 bit menus.

One possible approach would be make the updated depict key entries for the Best Res displays that leave out the 3 bit products, but immediately after those entries leave in the old entries, albeit commented out. If a site then decided that for a given level they wanted the 3 bit products to be in the Best Res display, they could just uncomment that entry and then rerun their -radar localization task.


Being proactive about problematic customization practices.

Throughout the discussions about how to implement dual polarization radar data, we have spent endless hours debating what seem like infinitesimally small nuances about the look and feel of everything. The point here is not to debate the merits of this approach. The point is to remind people that this did occur, and so everything we are doing with regards to the dual polarization stuff is based on the assumption that these infinitesimally small nuances are important.

One way we lose control of these nuances is if there are sites that are using less than optimal customization practices. As mature a system as AWIPS is, developers still occasionally get panicked e-mails from sites claiming that the system is broken, only to find that the code is working fine, and that less than optimal customization practices are the source of the problem.

For the most part, this is not a result of the current AWIPS focal point consciously using bad practices. Rather, it is usually because some previous AWIPS focal point got some bad advice and adopted a bad practice, and the practice continues due to inertia. This is especially insidious, because if you ask a site "are you following bad practice X?", the site will often answer that they are not, fully believing they are answering accurately, when in fact they ARE following bad practice X without realizing it.

Since the dual polarization stuff is dependent on controlling the configuration of both the radar area and the volume browser area, there are many bad practices that could hurt us. On the radar side, you have to worry about sites editing the nationalData/ radar key tables in place, or copying the entire default key tables to customFiles and then editing them. Also, some sites may have a customized version of the radar menus that they always just slap in place after an upgrade instead of merging it in. In OB9, there will be two versions of the menu they will have to do this with, so this is pretty important.

On the volume browser side, you have very similar issues. Editing the nationalData/ version of source, plane, or field tables in place, and copying the entire version of those tables into customFiles are the biggest worries, but sites that are rigorously following Dan Baumgardt's recommendations should not be doing either of those things. Also, there are many sites using the super wide so called "ingredients based" volume browser menus. If they are in the habit of just copying their ingredients based volume browser menu in place after an upgrade without merging, then we will lose all the additional CAPPI level stuff we have been working on.

Of course, we need to put something in the release notes cautioning sites about these things. However, because of reasons already stated, this may not be sufficient to avoid problems. One possible approach would be for GSD to put together a "bad practice sniffer script" that SST would run at every site and get the output back to GSD. If potentially problematic customization practices were identified, we would instruct the site's regional headquarters to specifically address some kind of official communication to the site, politely cautioning them about the potential problems.


Volume browser user interface complexity.

The complexity of the default volume browser user interface has generally grown with each release, and OB9 is no exception. That this complexity is growing is probably unavoidable, but there may be things we can do to manage it better. There are two possible approaches to this. One, create another category of sources, which would be those gridded data sources that primarily have data only at the earth's surface; this would greatly simplify the very long source menus. Two, change the VB user interface such that two rows of selectors are created above each major selection area; this would greatly reduce the width of the GUI without greatly increasing its height. This could be expecially convenient for those sites that use the "ingredients based" approach.


Should string keys be assigned to other data sets?

By implementing string keys in procedures for radar and volume browser displays, we are giving ourselves a great deal of flexibility for future keyspace management issues that may come up with those product types. Perhaps it would be reasonable to be proactive and also define string keys for some other data sets. Satelllite especially comes to mind as a data set that has a particularly convoluted keyspace that could benefit from this, but there would be no reason to necessarily limit such consideration to satellite.