The IEEE spectrum had an article last week about the FBI’s development of a new software platform called Sentinel. Sentinel is the FBI’s second chance at replacing one of their antiquated software systems. The previous attempt was called Virtual Case File which had the plug pulled on it in 2005 after 5 years and $170 million. Information Week also had an article about Sentinel here.
Large Engineering projects are difficult in general, but large software projects are even more so due to the flexibility and abstract nature of software. For example, design and construction of a new steel production facility is a large complicated project with many requirements, but it is possible to condense most of it down to a few root requirements. Possibly something like ‘Supply and build facility capable of producing x tons/yr of grade xxxx, y tons/yr of grade yyyy. Efficiencies shall be such that cost/ton of grade xxxx is less than $z based on our supplied costs in material and labor’, etc. There would be many additional requirements and specifications generated from this, but the project has a root requirement that is specific and measurable. The customer would develop a high level bid specification and ask qualified vendors/contractors to prepare a proposal. The qualified vendors would have previous experience with this type of project that would be used to develop budget and schedule estimates for the project.
Now consider the FBI’s root requirement which was something like ‘develop and supply replacement for antiquated software system.’ The root requirement is not very specific and not measurable. Obviously the vendors invited to bid on the project would work with the customer to place more definition and specificity around the requirements. The vendors would also have experience with this type of project and would be qualified based on past experience and their CMM rating. The problem is that the size and complexity of the existing system (and the root requirement) is multiple orders of magnitude greater than in the previous example. It would take both the vendor and customer an enormous level of effort (AKA time and money) to properly specify and document a complete set of requirements for a project of this complexity.
The typical scenario for these types of projects seems to be that the project is bid, budgeted and awarded based on incomplete and/or incorrect requirements. As development progresses mistakes and omissions in the requirements specification are discovered and additional requirements are added inflating the size of the project. This is called scope creep.
Project schedule, budget and quality are highly coupled. Only two sides of the triangle can be pulled. Budget and schedule always seem to be the most inflexible and therefore quality suffers. Requirements changes and scope creep were issues noted with both Sentinel and Virtual Case File projects.
The US government has lost an enormous amount of money over the years on numerous failed software projects. There have been voluminous post-mortem case studies developed about failed government software projects, yet failure still occurs. What can be done? (I’m keeping this really short)
In my opinion the (main) root cause of the problem is the reluctance of the large contractors to give up the waterfall model of development. Many Software Engineers (and managers) advocate “agile” development processes. Formal requirements generation is done away with in this approach. Instead, the customer works directly with the Engineers to develop small portions of functionality at a time. This ensures that the customer’s requirements are met. Normally 2 Engineers work together ensuring peer-reviewed software. Automated regression testing is usually a component of this process as well. The main criticism of this approach is the lack of high level focus on the overall system. The IEEE article notes that new FBI management elected to adopt agile development methods in an effort to increase quality and development rate. It appears to have made a positive impact, although maybe not as much as they had hoped for.
The FBI’s existing systems were developed in house by people that worked for the agency, were familiar with the domain and still took years. It is not realistic to assume that a replacement could be developed by an outside company with no domain knowledge in a much shorter period of time.
Large government software projects are difficult. If Engineers and managers could admit that it would be a big first step toward change and improvement. But it seems like many organizations want to continue doing the same thing and expect different results.
The Sentinel program is scheduled to be signed off on as officially functional on February 10, 2012.