March 27, 2000
One paradoxical aspect of computers and software is that the process of writing software, -- the price/performance of programmers, if you will -- has gained much less in productivity, compared to the evolution of computers and hardware themselves. Last month, in Part I, I talked about why programmers' productivity has evolved much slower than hardware. Now, let's continue this examination and explore some ways to significantly improve matters.
Out of the chaos of early programming came insights into better methods and structures for writing programs. From the 1950s through the mid 1970s, these insights centered on the program text, it's structure, layout, and partitioning. Even though the program could be well structured, large programs were difficult to write. Part of the problem was the huge number of details that each programmer had to remember, and part was because of the exponential growth of human communications as the program complexity increased. Something beyond program structure needed to be addressed.
Looking back on nearly four decades working with computers, certain elements of the programming process stand out as essentially unchanged. An obvious example is the knowledge and level of detail required to develop a program in C or C++. In some aspects, they compare unfavorably to using assembler. The syntax is more complex, the structures can be even more obscure than assembler-level data definitions and the potential hidden problems easily exceed those of assembler.
In return for the extra details of C and C++, we are supposed to get portability, readability, and maintainability. Sometimes we do, but it is well known that one can write obscure programs in many languages, even unintentionally. It isn't my purpose to make C or C++ a strawman to knock down. They fall in the class of languages that leaves every decision up to the programmer. It is this requirement that the programmer handle all of the details that characterizes the first generation tools.
A later generation of tools, still called languages, hide varying amounts of detail from the programmer and provide default routines that can do the job or be overridden by the programmer. For many simpler or one-shot jobs, the inefficiency of the run time process is inconsequential to the total cost when you consider the jump in programmer productivity. Well-known examples of this class of tools include Smalltalk, Perl, and Python.
A third generation of tools is focused and specialized, yet very powerful. Here, the appearance of traditional programming is hidden. The programmer deals on a functional level and all of the details are handled by the tool. Into this broad class fall Tcl/Tk, AWK, Bison, regular expressions, APL, Finite State Machines, BNF grammar processors, and VLSI logic processors.
None of these tools has eliminated the drudge work of general-purpose programming. Even the Interactive Development Environment (IDE), a standard of modern development, has not changed the basic requirement of in-depth knowledge and voluminous typing. It, too, is a first-generation tool. Creating a program should not be like an old-time garment worker who assembled suits one hand-cut piece at a time, yet this is the way most programmers still work, almost 50 years after Univac I was delivered to the Census bureau.
In a Unix class system, programming power comes from the ability to select from all three generations of tools and string them together to accomplish a task in the most expeditious manner.
This capability has been used since early days, and it is still one of the key elements to programming power. Even today, these tools remain isolated and non-othorogonal with varying I/O requirements. Thus the selection and assembly can only be done by a knowledgeable programmer. While some of the tools are past first generation, the assembly process is still first generation.
Here is the dilemma a programmer faces. To get good performance, details are important and take time to get right. Commercial software is time-to-delivery sensitive, and this requirement conflicts with building high-performance software. What really makes a programmer's job difficult and slow is the wide variety of programming techniques and skills in a typical team. This increases time spent by team members communicating details and resolving conflicts. Productivity is further aggravated by high programmer turnover rates that remove the trained people after they become most productive.
The core of the productivity problem is the level of detailed handwork required to specify a program. Multiply that by the need to coordinate every interaction between components and the task grows exponentially as the program grows. This problem also affects the other steps of documentation, quality assurance, and training.
Some help for the programmer has arrived with the use of concepts called Model-View-Controller (MVC), Objects and Patterns. But the lack of a unifying structure to make these concepts work together means that it must be done by people doing detailed designs and lots of handwork. Even with modern IDE, there are still too many detail decisions required to build a program.
Why? Colleges teach computer science with only secondary attention to the actual programming process. Since the science people are usually interested in solving challenging theoretical problems, the practical problem of improving programmer productivity gets little attention.
What appears to be missing is the engineering approach in the field of programming. An engineering approach requires a certain level of experience and knowledge of the industry techniques before working independently. This process is similar to the apprentice system in older crafts. Engineering can also be described as building on what works regardless of whether you know why it works.
The big change in programming discoveries started in the late 1970s. Prior to that, the discoveries and refinements dealt with details of how the program was written and structured -- the program conceptual space.
Starting with Model-View-Controller (MVC), this new approach worked on the higher level of problem conceptual space. MVC was introduced with Smalltalk by Xerox PARC in the late 1970s.
MVC abstracted the problem from the programming details, and then separated three aspects of the problem into independent but interacting components. MVC is briefly defined as:
Model: The data and logic that defines how to access and change that data. View: The way the data is presented to the user or another machine. Controller: Provides input and feedback to the model to change the data.
The power of this approach was that it basically did not matter how the individual components were created. As long as the interfaces were consistent, one could remove a component and replace it with something more powerful without impacting the other components at all. MVC was in some ways an early version of objects, but without the fine-grained nature of current object approaches.
MVC's acceptance suffered from three problems. First, it was a large leap to understanding the concept and how to make it work. Second, it was introduced as a part of the Smalltalk language that had its own significant learning curve. Third, the cost of training and conversion was a big obstacle to companies that considered the option. The costs were up front, the benefits were, maybe, down the road a ways. And maybe not. MVC was a powerful concept, but it did not make a big impact at that time.
Despite these negatives, MVC as a concept was a major turning point. MVC introduced ideas that were catalysts to the later complementary concepts of objects and patterns. You can view objects as MVC in small scale, and patterns as MVC in large scale.
Client/Server (C/S) in the 1980s, was the next big innovation, this time in the structure of the delivery mechanism. No longer was all processing done in a central point and only display and data entry done remotely. Now processing could be done where it was most appropriate, near the entry or near the data. However weak the original concept seems today, it did introduce the concept of doing the processing where the data was. Today, C/S is part of the solution and comes in two level (original flavor) and three level with database added.
Objects were the darlings of the 1990s. Objects introduced the concept of data capsules, that information relating to a single thing should be kept together and isolated from outside access except through well-defined interfaces. Objects take the MVC paradigm and implement it in smaller logical or functional units.
Patterns reflect the discovery of how many parts of different programs went through the same sequence of steps, even though the details and the data were different. A pattern is an abstraction of the larger scale structure of a program or group of programs. These patterns were seen as a structure that could be reused as a guide to building other programs that used similar steps.
MVC made a comeback in this decade. As objects became a mainstream idea, the value of MVC as a concept was rediscovered. Objects put a fine-grained structure around the smallest elements of a program, while MVC created a structure around the functional aspects of the program. Patterns identified a structure in applications.
The conceptual set of Objects, MVC and Patterns is a set of nesting concepts. Each element is made up from a collection of smaller elements, starting with objects. There is an amazing analogy to the physical structure of the universe that starts, we think, with elementals like quarks and leptons, making particles that make atoms, molecules, planets, solar systems, galaxies, and universes.
In an analogous fashion, I see that programs are built from bits, bytes, data elements, objects, MVC, and patterns, each structure built upon the prior structure. This hierarchy of structures is a concept that can be used to facilitate the creation of programs by automating the building of a program by assembly of building blocks
How would this concept work? Think of objects as the Leggo building blocks of a program, and MVC as the walls, floors, and ceilings, using Patterns as the design of the buildings. Then consider C/S as the way buildings are connected into a factory. Two new tools are needed to make this possible. One is a way to specify the interfaces, the way the blocks, walls, and buildings can be connected. The second is a tool analogous to a crane that can stack and assemble the pieces to build the complete structure.
This first tool is a Meta Interface Language (MIL) that describes the component interfaces in a way that lets components be automatically found and assembled, either by a person or program. If such an MIL is created and the required components for a program described, then programming would lose much of the detail dogwork that is involved, along with the human errors that are made in the process.
The second tool is one that programmers would use to describe the function and structure of the desired program. This tool would inspect its database of objects, MVC functions, patterns, and the MIL that describes how these can be connected together. It's a software version of the VLSI electronic building blocks, and the MIL is analogous to the VLSI design languages. The building tool is analogous to VLSI design tools. I call this the Program Structure Builder (PSB).
The PSB follows the operator's directions in selecting and assembling the necessary components to create the program. It takes components with their MIL descriptions and assembles them into functions. All details will be hidden unless conflicts arise or components have to be created or modified to meet special needs. The operator (programmer) may use block-function diagrams, finite state machines, or other descriptive technique to specify the sequence and the results, but not the details of the process. This will finally raise the programmer from a piece worker to assembly line manager, with the PSB carrying out the commands.
Creating the MIL and constructing a PSB is almost certainly beyond one person or a small group. New tools, standards, and programs must be developed to automate the detail work of programming. The routine of typing in every character in a program one at a time is far too slow and error prone. The difficulty of good internal design for a program should not be left in human hands, but given to programmed tools.
PSB does not replace the programmer, it raises the level at which the programmer works. The upside of such an approach is that programmer productivity would rise substantially, by large amounts for common tasks. Errors would diminish by similar amounts because of reuse of objects and elimination of interface mistakes. More complex jobs could be undertaken by smaller teams, with faster results.
One of the best examples of a part of PSB capabilities is the programming system called Python. See http://www.python.org for access to the latest programs and libraries. At first glance, there seem to be no downside to such a tool. There will be speed bumps on this road to programming nirvana. Starting with the development of a new MIL for the type and interfaces of the components, new tools to build, analyze, and test existing and new components for compliance.
It is likely that a lot of current components will have to be rebuilt or scrapped in the light of the new tools and approaches. Conversion tools will be important for a decade or more as older programs are replaced. Training will be a long-term effort, and getting programmers who are addicted to tweaking every bit and byte to let go of that level of detailed control are two human problems. How long it will take to remedy that is unpredictable.
Even more, it will take some courageous souls to step up to the plate and take a swing at dramatically changing the way programming is done. It is time to grow out of the craftsman era of programming. Construction-engineering design is based on form and function, using details like strength of materials, cost, and availability for selection of components.
Programming construction should move toward the engineering concepts of form and function and away from the hand design and building each piece of code. Craftsmanship belongs in the artistic endeavors of design, not in the detail work of joining bricks with mortar, or lines of instructions.
Some early steps in this direction can be found in the IBM San Francisco project and Java Beans; and with Proposal Based Architecture from Kinexis and the Misunderstood Article.
All content on this site is Copyright 2001 by Bill Nicholls