Copyright 1999, 2001 All Rights Reserved
Building powerful software is hard. It is a complex, demanding process which requires organization, teamwork and communications. The skills and resources involved are not trivial. If you understand the difficult challenge of the task, you will be less surprised that one third of all major projects fail. They are usually stopped after big financial overruns, sometimes years after the original optimistic completion dates have been missed.
Why does this happen? Why does management allow this to happen? Those two questions are often asked, seldom answered. The reasons, like the projects, are complex. Human nature being what it is, nobody likes to investigate failure too closely lest they become associated with it. Worst of all, rarely does anyone learn enough to avoid a repeat.
What I am writing here is my approach to avoiding this disaster. I have been on the inside of several ambitious projects, some of which failed despite the best efforts and copious overtime of the programming staff. They couldn't fix the problem because they had no leverage over the cause. The cause starts much higher than the programming staff or manager. The cause starts at the very top of the organization.
It isn't stupidity or carelessness, bad attitude or deliberate failure. Very few executives set out to fail. They simply don't understand the intensity of the requirements - organization, teamwork and communications. This lack is also prevelant at the managerial levels as well. Everybody in management thinks they understand the requirements, but few really do.
The reality of a big software project is far more complex than it appears, and even more interconnected than most practitioners suspect. The classic iceberg metaphor is appropriate here - management sees the tip and plans accordingly. Their ship of software flounders on the hidden part beneath.
Why do I claim to know the answer? In a word, experience. I started programming 40 years ago and made my living at it for more than 25 years. Since I moved to consulting and writing, I have continued to study the causes and potential cures for this problem. I believe I have a valid approach to making the process work, though not a guaranteed one. Once you read why the problems occur, and what the causes are, you will find the solution understandable.
I've mentioned the main issues - organization, teamwork and communications. A successful project also requires adequate resources in time, equipment and competent staff. Without proper resources, no project will succeed. What is proper and required will be covered as it comes up.
Teamwork and organization can speed or delay success by a factor of two or more, depending on details. How best to set these up for a project will be covered in some detail. Here's a hint - good project organization has a tight communications ring.
In summary, while most companies have some teamwork and adequate organization, software failures can most often be classed as communications failures. These failures happen at and between every layer of management, and the more layers, the bigger the probability of failure. In short, complete and comprehensive communications are a necessary, but not sufficient, requirement for software project success.
Traditional companies have evolved from having the programming manager report to accounting to having an IT (Information Technology) expert as part of the executive team. This recent promotion belatedly realizes that information is a core component of every business. What was once optional is now necessary.
Software projects must be organized (not managed) from the executive level. Only at the executive level can all the potential connections and tradeoffs be evaluated, broad requirements determined and sufficient resources allocated. Only by realizing that these needs will change over time can broad design goals be established that will flex with that change. What happens at this level is strategic planning for business advantage. No mention of bit or byte is needed or proper.
Let me be specific about the executive level. This is where business strategy is established, market penetration planned and corporate goals set. The IT executive has a dual path at this level. He or she contributes to strategy and planning by providing historic and trend information to the executive team, and also tells them what is and is not currently within their capability from an IT point of view.
When the strategy and goals are set, the IT executive shifts into IT strategy and tactics to develop the infrastructure to support the corporate goals. This is a critical point in the whole process. This is the starting point for a whole generation of infrastructure and application development. It need to be approached from a clean slate. Possibly the worst approach would be to see how to 'modify' the current system to meet the new goals. This is a recepie for disaster.
There are good reasons why this doesn't work well. First, because the process 'looks easy', the budget and timeframes are almost sure to be less than is needed. Second, starting from the old system creates a mindset that fails to look deeply into the new goals. Third, the inevitable subtle differences become the reefs upon which the original plan flounders. Aggrivating these mistakes is the tendency of management to assume that throwing more time and money at the problem is a cure.
It isn't. This is like throwing gasoline on a fire. From Brooks "The Mythical Man Month," adding manpower to a late project makes it later. The conclusion that is almost never made is that the whole design approach of modifying the current software is wrong.
Why is this so? It is because the typical design of an application system, particularly databases, is done in response to a specific set of objectives and yields the best design *for that set of objectives*. Rarely is management willing to invest in a system that is designed to capture all the information, even if it is not currently needed or a use for it forseen. Yet this is just the model required to be flexible enough to respond to future changes in strategy.
The result is a blivit, defined as seven pounds of dirt in a five pound bag. Management insists it can be done because it is possible. They refuse to hear that it is the wrong approach because the right one seems too expensive. As a result, the programming team laboriously modifies the current system to provide new features. This is a classic example of the problems brought about by modifying existing code that was designed under a different structure.
The word 'kluge' jumps to mind. Instead of working from a clean design, people apply crowbars and sledgehammers to bend the old design into something close to the new. The modified code becomes more complex, bugs creep in and the whole application becomes fragile. Management blames programming, and vice versa, but neither really knows the real reason for why it happened that way.
As I wrote above, the correct way is to start with a clean sheet of paper and build a new data flow, databses and applications. Sometimes this new view will clean up old problems which were known but could not be fixed.
This approach has several advantages. First, it gives a clean view of the new system and can be understood to meet the new strategy objectives. Second, it does not impact development and maintenence on existing applications. Third, the new design can take full advantages of new technologies and programming tools.
This new design does not have to bend because of old software, and that enables faster programming. It is often easier and faster to create a whole new program than to dig into old code someone else wrote and figure out how to change it. It is almost always faster to debug new code.
The most important aspect of this approach is the opportunity to build the new system with a clear definition and good communications. Much of the definition and communications requirements are spelled out in the four part "Building Software in an Organized Fashion" (BSOF) series of columns I wrote for Byte.com.
I'll be revising those columns with more detail and better explanation after I write the one missing piece on managing projects and programmers. This is the missing link between executive strategy and the actual carrying out of the process that creates the applications that are needed. Traditional techniques of motivating and managing programmers work poorly because of a serious mismatch between corporate incentives and programmer motivation. Both sides are dissatisfied with the result. There is a better way.
All content on this site is Copyright 2001 and 2002 by Bill Nicholls