A Proven Software Configuration Management System for Workstation Development

1. INTRODUCTION

The FX-Advanced system has been under development at the Forecast Systems Laboratory (FSL) for several years. The great success of this development is reflected in the National Weather Service's decision to incorporate the software into its AWIPS system. FSL's software configuration management has played a key role in the success of the FX-Advanced system.

The FX-Advanced software development process uses multiple interactive and incremental development cycles as detailed in the spiral model for software development (Boehm 1988). The key issue of this software development model is the high frequency of software change. To handle this challenge, the software configuration management of the FX-Advanced system uses the check-out and check-in model (Feiler 1991) built on the Baseline Configuration System (BCS) and its underlying version control software, Revision Control System (RCS), that provides the actual check-out/check-in functionality. The advantages to using BCS are its simplicity, ease of use, and low cost.

This paper covers the major operational aspects of the FX-Advanced software configuration management process, including identification, control, status accounting, review and audit, problem tracking, and the release process. This practical case study shows that the FX-Advanced software configuration management process is quite promising and is very applicable to other similar software development projects.

2. THE FX-ADVANCED SOFTWARE CONFIGURATION MANAGEMENT TOOLS

The FX-Advanced uses in-house development tools based on public domain software to perform the software configuration management tasks. Although public domain software has limited capabilities, it is cost free and if chosen properly, very stable. In-house development tools may not provide full software configuration management functionality, but they can satisfy many needs and are easily customized to address requirement changes in software configuration management. The tools used to manage the FX-Advanced software include:

BCS: Provides software management in a centralized repository called the baseline. It creates and maintains a local staging area in each developer's home directory which is linked to the baseline via the UNIX symbolic link mechanism (Berkenbilt 1994).
RCS: The underlying version control software used with BCS. RCS provides file check-out and check-in operations, file revision control, and some basic administrative operations.
Bc-scripts: The intuitive and easy to use front end script commands for BCS developed at FSL which simplify the long and tedious BCS commands.
GNU make: Provides standard "make" functionality and some very useful extensions. Its "dependency mechanism" is one of the keys to software identification at the file name level.
Tkgnats/gnats: Provides the user interface and database management for problem tracking, including tracking change requests and problem analysis, and logging problem and bug fixes. It also integrates with the UNIX mail system to provide a convenient way to notify all affected persons of problems in the tracking process.
E-mail: Provides a communication channel for the entire project to use for general notices, bug and problem reports, warning notices, group discussion, information and technologies exchange, etc. The use of e-mail saves a lot of time in routine communications and task coordination.

3. SOFTWARE IDENTIFICATION

The purpose of software identification is to identify the software structure, its components and their types, and to make these components unique and accessible in some form (Dart 1991). Software identification for the FX-Advanced can be distinguished at the following levels.

Software tree name: Based on the spiral model for software development, a new FX-Advanced software tree is created approximately every 2-to-4-months. The new tree name reflects a major milestone or the expected release date.
Subsystem components: The FX-Advanced system components were carefully assigned to subdirectories during the system design phase. For example, "ipc" represents the interprocess communication component, and "igc" the interactive graphic capability component, etc.
Source files: The lowest level of software identification in the FX-Advanced system is the source file. All of the source file names (including data files, C++ class files, C files, FORTRAN files, and Tcl/Tk, perl, and shell scripts files) are described in the GNU makefile for each subdirectory. Also the source files themselves are identified by the RCS "$Id$" inside the source files. For example, the file name, version, modification date and time, and user ID for a C++ header (.H) file, are readily identifiable by the $Id in the following example:

static const char* const CGMGraphicDepict_H_Id = "$Id: CGMGraphicDepict.H,v 1.1 1996/12/09 23:46:12 mfchen Exp $";

4. SOFTWARE CONTROL

The changes and releases of a product throughout its life cycle are carefully controlled. This control ensures software consistency with the baseline product (Dart 1991). The FX-Advanced system relies on BCS to control changes to its software (Fig. 1). The baseline software tree is located on a designated baseline workstation. To modify the software, individual developers create their software tree on their local development workstations, known as the local staging area. By creating symbolic links to the baseline at the time the local staging area is created, all developers share the centralized baseline tree and files. When developers want to work on FX-Advanced software, they check out and lock the desired files. This entails "staging" the files by copying them to the local staging area and replacing the symbolic links; the files are also locked for editing. After the development and unit test work is completed, the developer checks in the files by "unstaging" the files and replacing them with symbolic links.

For control and manipulation of the source files, developers can use either BCS commands or bc-script commands to check out (stage) and check in (unstage) files. Bc-script commands, the front end of BCS, provide easy-to-use, more intuitive commands. For example, users can use bc-newfile, bc-lock, and bc-release to create new files, check out files, and check in files, respectively.

When a developer creates a new file and adds it to the baseline, there is no symbolic link to it from the staging areas of other developers, who are now unsynchronized with the baseline. Hence, developers must run the BCS "sync_staging" command or the bc-script "bc-sync" command frequently to keep their staging areas synchronized with the baseline.

To allow control of the baseline software, the developers are asked to check in files only during two time periods: 11 am - 12 pm and 4 pm - 8 pm. Developers can run software builds outside of these times and be assured that the software will be consistent for the duration of a build. A nightly software build is executed to ensure that software problems and bugs are detected as soon as possible.

5. SOFTWARE STATUS ACCOUNTING

Software status accounting refers to the administrative tracking and reporting of all software items that are formally identified and controlled (Bersoff 1984). Basically, there are three software status accounting activities in the FX-Advanced software configuration management process: the source files logs, the daily modification report, and the file status report. It is extremely helpful to use RCS as the underlying control software since it provides a logging mechanism for each revision. Whenever a source file is changed, the reason or purpose of the change can be logged in the RCS file. This information documents all changes to the source file and makes the source file traceable. In addition, with RCS's revision control capabilities, obsolete files or files checked in by mistake can be removed or reverted to the correct revisions. The RCS "rlog" command can be used to inspect the logged information.

In the FX-Advanced software development process, a nightly software build verifies that the previous day's development compiles error free. This automatic build is started via "cron". The status of each action performed by the build is logged. After the build is complete, an automatic process scans the log file and extracts the warning and error messages. This summary is then e-mailed to developers and project managers. If there is an error, everyone will know about it by the next morning and correcting the error will become the top priority for the affected developers. This process guarantees finding and fixing software problems as soon as possible.

At any time, a developer may want to determine which files are locked and who locked them. The bc-script "bc-status" can be used to get the lock status of files in a given directory and its subdirectories.

6. SOFTWARE AUDIT AND REVIEW

The purpose of the software audit and review process is to validate the completeness of a product and to maintain consistency among the components (Dart 1991). The FX-Advanced software audit and review process includes:

Code walk-throughs: For critical software components, a code walk-through may be conducted to help developers find bugs at an early stage. Code walk-throughs also ensure that developers follow the project's standards and conventions (e.g., coding conventions).
User Interface Working Group (UIWG): The FX-Advanced system development cycle includes weekly UIWG meetings to discuss the major aspects of the FX-Advanced user interface. Through discussion and critique, the UI can be refined towards optimal user efficiency. Furthermore, the participants benefit from the extensive experience of the senior staff who help with the software audit and review.
Integration test: After the developers test the completed components, and there is no error in the nightly build, the software may be "pushed" to the integration and test workstations. There, the software testers perform the integration tests on the new software components to see if they meet established requirements.
Internal usage: There are many forecasters at FSL who use FX-Advanced regularly for the daily weather briefings. New software is made available to these users periodically. Through the daily use of FX-Advanced by forecasters, the stability, reliability, and usability of new software components can be significantly improved.
Experimental field test: The Denver WFO is the experimental field test site for the FX-Advanced system. Through such intensive field use, especially in severe weather situations, most remaining problems and bugs can be detected and dealt with by the problem tracking process. It is at the WFO where the FX-Advanced software's stability, reliability, and usability are truly verified and validated.

7. SOFTWARE PROBLEM TRACKING

Change management support is prevalent in current commercial software configuration management tools. As Dart (1996) mentioned, developer-oriented and process-oriented software configuration management tools support change management. One approach to change management is software problem tracking.

Software change control is not an easy task in any software development environment. The tkgnats tool is used to help manage software problems and to track their status. The FX-Advanced system problem tracking concept is shown in Figure 2 and its work flow is described below.

Problem detection and reporting (step 1 & 2): Through integration tests, internal user exercise, and experimental field user tests, some problems are detected and collected in the tkgnats database.
Problem notification and task assignment (step 3 & 4): When a problem report is created, technical and CM managers and group leaders are notified automatically by tkgnats via e-mail. They evaluate the problem and assign it to a developer. Tkgnats then notifies the developer.
Query and correction of problem (step 4 and 5): Upon notification of the task assignment, the developer queries tkgnats for the problem report using the user's ID. With the problem description, the developer evaluates and corrects the problem.
Problem Analysis (step 6 & 7): After the developer fixes the problem and describes the fix in the tkgnats problem report, the developer changes the report state to "analyzed" and forwards the fix to the SCM managers and testers. Again, tkgnat notifies all affected parties.
Verification and Validation (V&V, step 8 and 9): At the appropriate time, the software is pushed to the integration and test workstations. The testers may now commence with the verification and validation. Once the software is released to the experimental field site, the field users further verify and validate the problem fix.
Feedback: After V&V, if the problem is still there or not completely fixed, the entire problem tracking process may need to be repeated.
Closing: If testing and V&V is successful, then the problem case is closed.

8. SOFTWARE RELEASE PROCESS

When software is released to a large number of users, management of the software release can be quite complicated. A lot needs to be done, both at the development site and at the user sites (Hoek et al. 1997). FSL's software release responsibilities for FXA are limited to internal users, the experimental field site (Denver WFO), and NWS head quarters. Hence, its software release process needs to accommodate only these three sites. A new version of the FX-Advanced software is released to internal users on a weekly basis. When the software build is error free, it is pushed to the integration and test workstations. The internal users at FSL use the most up-to-date software for the daily weather briefings. Through this use, the critical problems and bugs can be found quickly.

The FX-Advanced software is released to the Denver WFO experimental site when the baseline's requirements are met and the critical known bugs are fixed. The Denver WFO forecasters use FX-Advanced to produce public weather forecasts. Therefore, this experimental site puts FX-Advanced through intensive stress tests, especially in severe weather situations. This experimental field test coupled with the problem tracking process allows the FX-Advanced system to become increasingly more stable and reliable as more problems are found and fixed. After about one month of field testing, it is released to NWS. At this point the NWS integrates FX-Advanced into AWIPS and starts it through the more complicated AWIPS software release process.

9.1 Advantages vs. Disadvantages

There are several advantages and disadvantages to the current FX-Advanced software configuration management process. The advantages include low cost, a direct result of using in-house development tools which are based on public domain software. A second advantage is the flexibility provided by in-house wrapper scripts which allow FSL to make changes to the process to meet developers needs. Finally, the process is ease to use. The bc-script commands provide the developers with a simple and intuitive front end to BCS, with almost no training needed for new users.

However, there are some problems with the current FX-Advanced software configuration management process. First, since the software does not enforce adherence to the process, managers and developers sometimes do not follow the proper procedures. This has resulted in inefficient software development and management of certain pieces of code. Second, the current FX-Advanced documents are not under BCS version control. This lack of document version control can make it difficult to properly correlate software documentation with the associated code.

9.2 Proposal for Improvements

Although the FX-Advanced software configuration management process runs smoothly and works fine within FSL, several areas can be improved. First, more formal working procedures and guidelines should be defined, concentrating on the critical processing steps. This should improve software quality and reduce development costs. Second, the status accounting could be more robust. Compared with commercial configuration management tools, FSL's in-house status accounting tools are rather weak. For instance, FSL should develop a tool to track which files are checked out each day to save time in finding problems and bugs.

9.3 Future Challenges

Based on FSL's plans for FX-Advanced, there are some software configuration management challenges that FSL is now facing. First is the management of multiple baseline trees, one for each release, and another for the next release in which most development is occurring. Changes must be made in older trees to fix problems found in the associated releases, and these fixes must usually be made in many trees. As trees continue to proliferate managing this complexity gets more difficult.

Second, FSL plans to integrate several projects into the FX-Advanced system. In this sense, the challenge is not only integration of the software but also integration of the software configuration management process.

Third is the need for a distributed software configuration management process (Hoek 1996). AWIPS developers at other organizations need to be able to modify the FX-Advanced. FSL has recently acquired PCMS, a commercial configuration management software management package, which was selected by the AWIPS contractor, to help resolve this problem. PCMS is now being used for all development specific to AWIPS; however, BCS will continue to be used to manage software for other FSL tasks.

The last challenge is the management of FX-Advanced software releases that are independent of AWIPS. Organizations such as the FAA and the Taiwan Central Weather Bureau are interested in adapting FX-Advanced for their own purposes with FSL's help. This requires making it separable from AWIPS while maintaining it as part of AWIPS, a very difficult challenge indeed.

11. REFERENCES

Berkenbilt, J., 1994: BCS: A Baseline Configuration System, BCS Version 2.0.2 document.

Bersoff, E., 1984: Elements of Software Configuration Management, IEEE Trans. Software Engineering, Vol. SE-10, No. 1, pp. 79-87.

Boehm, B., 1988: A Spiral Model of Software Development and Enhancement, IEEE Computer, Vol. 21. No. 5, pp. 61-72.

Chen, M., 1998: The Software Configuration Management of the WFO-Advanced System: A Practical Case Study. Software Engineering project paper, Computer Science, The University of Colorado, Boulder, Colorado.

Dart, S., 1991: Concepts in Configuration Management Systems, In Proceedings of the Third International Workshop on Software Configuration Management, ACM SIGSOFT, pp. 1-18.

Dart, S., 1996: Not All Tools Are Created Equal, Application Development Trends, pp 39-47.

Feiler, P., 1991: Configuration Management Models in Commercial Environments, Technical Report, CMU/SEI-91-TR-7, Software Engineering Institute, Pittsburgh, Pennsylvania.

Hoek, A. & Heimbigner, D. & Wolf, A., 1996: A Generic, Peer-to-Peer Repository for Distributed Configuration Management, Proceedings of the 18th Inter. Conf. on Software Engineering, Berlin, Germany.

Hoek, A. & Hall, R. & Wolf, A., 1997: Software Release Management, Proceedings of the 6th European Software Engineering Conference, LNCS 1301, Springer, Berlin.