February 28, 2000
Despite several generations of rapid hardware advances since the 1960s, software has barely ambled through one generation. The classic description that the "Cobbler's children have no shoes" can be applied directly to programmers.
While programmers have diligently built end-user castles, they are still using stone-age tools based on hand labor. Recent developments with objects, components and patterns have opened the potential for quantum jumps in programmer tool capability and productivity. What will it take to make this leap?
The search for better ways to solve problems using computers has very slowly led to discovery of more effective programming methods. The small size and relative simplicity of early 1960s programs made success easy. This led to the false assumption that difficulty increases linearly with program size. Later experience showed difficulty was exponential based on the number of interactions within the program.
What took longer to become clear was that there was no "silver bullet" to kill the problem of programming complexity. Early efforts with structured programming reduced this problem, but added a new set of interactions. Each additional technique added some power at the cost of additional training and new ways to go wrong.
The new methods of objects and patterns differ from their predecessors, yet the latest tools do not seem to herald the dawn of a new programming age. In part one of this column, I will show the evolution of methods from the dark ages of spaghetti programming through the discoveries of subroutines, structure, project organization, and programmer teams. This historic approach will make the slow pace of progress clear to those who have not lived through it.
In part two, you will see a major change in methodology with Model-View-Controller (MVC) methods, then Client-Server, Objects, MVC again and Patterns. Still, even today we are well short of what I would call a programming revolution. What is needed is a quantum jump in programming methodology. I'll venture some analysis and suggestions on how that could happen at the end of part two, next month.
The Dark Ages In the beginning, the mid 1950s to early 1960s, there was chaos. There was no formal study of programming, no university degrees in computer science, just people trying to solve problems that were beyond a roomful of calculators.
Programming began with writing code in decimal numbers, positioned in specific locations in the computer's memory. This was at the lowest level- direct entry into the computer memory prior to starting the computer at the first instruction. I used this technique with an IBM 1620 at the University of Notre Dame. One instruction to read the first card into the reader, another to jump to the first location of the card buffer. Booting, 1963 style.
Fortunately, my school had experienced people who had extended the Fortran compilers. Professor R. S. Eikenberry, an aerospace engineering professor, along with others had developed a load and go Fortran compiler. The original Fortran, not Fortran II or IV. It was called 'DoAll Fortran.' The compiler resided in memory, read control cards and compiled and executed the student's programs, then punched cards for output. The student carried the punch deck over to the IBM 407 and listed them out on the traditional green bar paper. It was a major productivity advance over the previous method of teaching students programming.
Describing the programming style of those days as 'ad-hoc' is accurate for most and flattering to some. Whatever worked was good, what didn't was bad. Specifications? Here's a sheet of paper torn from the professor's scrapbook. Testing? It compiles and runs the single hand-checked example -- it must be OK. The only way to look back at those days calmly is to understand the scale of the jobs was much smaller, and many could be handled by one person in a few weeks to a few months.
Student programs were another ball of twine entirely. Fortran, with GoTo and the suicidal Computed GoTo, was the source of innumerable spaghetti threaded programs. Modifying one of those was an exercise in masochism. They were tottering structures of glass, balanced in the tip of the pinhead that wrote them. In those early days, few people knew better methods and nobody was talking about 'computer science.' By analogy, an untrained person with a hammer and saw had a better chance of building a house that stood than these programmers had of building a solid program.
I remember it well, I was one of them. To a great extent, we were saved by the limited scale of the equipment that was available. Big jobs were broken down into several runs with data written to tape and read back in. The computers limited the size of the programs, which forced a form of partitioning, though with little organization. A big system had 64K words of 36 bits, a drum for OS and small scratch files, and slow tapes for everything else. Most projects those days fit easily into the system, and on the multiprogramming OS, we could run three or even four small tape jobs at the same time. Wow.
The best known early paper that addressed building programs was "Programming Considered as a Human Activity" by E. Dijkstra in 1965. Long before most of us had a clue on how to create good programs, his five-page paper spelled it out clearly. Actually doing what he suggested was rather more difficult, but concepts like elegance are still essential to building excellent programs. This paper is so clear that it rewards reading time after time.
In 1967, another paper, "GoTo Considered Harmful" by E. Dijkstra is probably the best-known attack dealing with specific problems of programming large jobs. By then, some systems had a megabyte or more of memory and were big enough to enable programmers to machine gun themselves in the knees. For many, it was a difficult transition. Avoid GoTo? How?
Subroutines became the answer for all such problems. The early uses of subroutines were restrained and effective, clearing the way for larger programs to be written. Soon subroutines proliferated and, like cockroaches in a dirty kitchen, became the source of another set of problems. There were no standards for choosing what to extract for subroutines or how to pass parameters.
A form of shared data storage known as Common was used extensively. This enabled further growth in program size. A Common definition of variables and arrays was created in the main program, and each subroutine needed to define one exactly like it in terms of variable type and size, but alas, not in name. Common proliferated, and became Named Common and plain Common. Names inevitably differed in the subroutines, from the main program, and from each other. Another level of chaos had developed. Programmers developed their own standards for using Common.
Where multiple programmers collaborated, as was becoming typical on large projects, someone was chosen to manage the interconnection between subroutines. Common gradually came under control as a standard deck that was duplicated and placed in each programmer's box of cards. The cards were another hazard for programmers. Dropping a 2000 card box was a major disaster. Not only did you have to put the cards back in order without sequence numbers, but then it took a pass or two through the compiler to work out the bugs caused by your missing a few misplaced cards. A dropped box was typically equivalent to three days lost time.
Those programmers who have worked in the field less than 25 years probably haven't seen a punched card. Cards were the ubiquitous form of source and object code, developing a whole sub-industry for supply, from key punches to card readers to storage cabinets. Slightly larger than a dollar bill, it was made of paper to exacting requirements for dimensions, stiffness, and strength. Numerous companies jumped into the supplier ranks with less-expensive supplies, and while the card remained the primary storage for programs, made significant profits on a commodity product.
The next step in the evolution of programming methods came with the introduction of structured programming (SP). This concept was described in a paper by, you guessed it, E. Dijkstra titled "Structured Programming." It was printed in a NATO sponsored Science Committee report in Italy, 1969. This eight-page paper succinctly describes the concept, a program "As a string of pearls" created by carefully crafting modules into a whole process. He was way ahead of his time in 1969, and 30+ years later, still ahead of quite a few practitioners.
This was a significant step in that it was the formal recognition of the importance of design in computer programs, and supplied a methodology that produced improved designs. At the time this was seen by others as the answer to programming problems. Now any project could be built successfully, and many of the early uses of SP worked well. As with all tools, exploration of its limitations was not long in coming. SP let larger and more complex programs be undertaken, which was enabled by larger and faster computers. Once again, the process created its own downfall, a kind of evolutionary death caused by hypergrowth.
Even with those problems, the programming community was developing techniques to handle larger and more complex programs. Despite the weaknesses of SP, it offered another step in greater complexity.
The failure of SP to be the answer to all problems also caused a recognition that the whole programming process was far more complex than had generally been realized. A broader perspective on the whole issue of delivering complex software, its perils and pitfalls, arrived in an insightful book, The Mythical Man-Month by Frederick P. Brooks, referenced in my columns a number of times. This book is important for a number of reasons.
It was written by a man who had full responsibility for controlling the development of IBM's flagship operating system, OS/360, in all its versions -- PCP, MFT, and MVT. The book contained things to do and to avoid, and honestly reported numerous lessons learned the hard way. The book taught lessons at all levels from beginning programmer through upper management, and had a substantial impact on practices in the industry. It also stimulated further research into solving these problems.
Most importantly, this book looked at programming outside of the academic world and placed it in a business environment. No longer was programming looked at as simply a problem-solving exercise. It was clearly placed as part of a business organization that had to meet such annoying requirements as schedule, function and reliability. Programming was now part of a system and the process had to expand to deal with all of the human and business issues as well as the technical ones. This large rock dropped in the pond of programming has generated ripples we see even today.
Another paper, "Chief Programmer Team Management of Production Programming" by F.T Baker, explored the approach of building a support team around one super-programmer. This person would be responsible for delivering the most difficult code. Other programming tasks could be done by mere humans, who were in turn supported by tool builders and librarians. It was a powerful concept that stumbled on human hubris, as most programmers consider themselves to be in the super class.
But the team concept and supporting people and tools have become a standard part of the programming environment. Teams rapidly evolved into small groups of programmers with a team leader, who might not be the best programmer. Work was usually aligned with skills, and good teams with internal support between members could create working programs very effectively. But when the job required multiple teams and multiple years, communications problems and turnover created invisible monkey wrenches in the works.
Another level of organization had to be added to manage large projects, with project managers. Efforts at using planning tools were unreliable at accurate forecasting because many projects broke new ground for the participants and their past experience was not effective at providing good estimates. Changing specifications in mid-stream, tools with hidden bugs, unclear communications, all took their toll on productivity. It seemed that even the team approach had reached its limits by the mid 1970s.
In the next column, I will describe the Model-View-Controller (MVC) concept and how it differed from earlier concepts. Then the development of client-server, the creation of objects, MVC again and finally, patterns. After discussing the implications of what we have learned in the past 40 years, I'll suggest a few possible steps for the future.
All content on this site is Copyright 2001 by Bill Nicholls