The basis for this new world will be the combination of Open Source software and Open Standards - which I call the "Open Software Environment (OSE)." This combination has already created multiple open implementations for Digital Libraries, which I wrote about in my November 2002 column for Byte.com.
Despite the multiple implementations and different objectives, almost all these projects adhere to the basic XML, DMCI and OAI standards that enable interoperation and harvesting at the index level. Thus the card catalog of the future is already here, on line and covering millions of documents.
The startling part of this is that most of this work has been done since these standards were ratified a few years ago. The success of Digital Libraries (DLibs) in this relatively short time represents the ability to conquer software projects that until now have been too large or complex to consider.
Digital Libraries are a tough problem, involving multiple document formats, multiple languages, a database for storing documents, a search system that can scale to millions, indexes with at least the ten basic DMCI entries, with hundreds of key extensions for specific subjects. Multiply this by thousands of Dlibs and thousands of subjects and the scale of the challenge becomes clear.
The OSE also represents the kind of environment we could have had if the Microsoft monopoly had not smothered the software industry. Despite the weak set of sanctions put in place by the US Justice Department, they appear to be enough to prevent MS from stomping out the OSE by using its current monopoly in desktop OS and office suites.
The OSE's strength comes from the Internet, not from selling software. This means that the environment where the OSE grew was free enough from MS interference to grow very fast. Microsoft's later attempts to turn the Internet into a private fiefdom with Frontpage and Internet Explorer extensions failed because the early OSE could progress faster than MS could exploit any opportunities.
In broad terms, the Internet has already given us a cornucopia of useful software. Efforts like Linux and the BSD tribe supply powerful operating systems, the Star Office and Open Office address the office suite needs, and many open languages and powerful tools (Apache, Tomcat, PostgreSQL) simply skim the surface of Open Source's growing software library.
This is only the beginning. Most of these projects develop their own standards as they build, or use existing standards. To this date, the only projects to be developed on openly created new standards (that I know of) is the Digital Libraries environment.
It is the development of Open Standards, cooperatively created to solve a general problem, that represents the key to OSE success. It is our new freedom from preload monopoly domination, and the existing open operating systems that will enable an explosion of OSE developed software.
New software, developed with internal XML based representation, will create a revolution in knowledge discovery, changing science, technology, business, education, medicine and people's lives.
This is because XML incorporates both structural and content information in a standard way, enabling the category of the information to be extracted along with the content. Once category is incorporated with each and every unit of content, knowledge discovery systems can parse the XML information to derive knowledge.
There is still a step beyond knowledge which humans will supply for a long time to come. That is wisdom, which I wrote about in a column named "From Data to Wisdom." If the knowledge systems can bring more effective knowledge to us, perhaps we will be able to show more wisdom.
The implications of these developments will go beyond what I can write here simply because I cannot even guess at many possibilities. What I write is based on certain assumptions listed here:
Let's start with a major US problem today - education. The current system struggles to graduate highschool students with bare literacy and basic math, spending too little time on the essentials. Not all schools fail, but the problems are obvious to any who look without bias.
Knowledge based systems, customized for education, can offer each person individual instruction paced to their speed. The teacher becomes a mentor, giving help and guidance where needed, relieved from forcing all students to the same learning pace or style.
Students are able to indulge their curiosity, which is nearly impossible in the current system. It isn't computers that will save education, it is the knowledge software and retrained teachers that will change education. The computer is only a delivery vehicle, and like cars and trucks, will morph into a wide variety of forms, specialized for many different uses.
Schools may see a touch sensitive flat screen, built into the desk with virtual keyboard when necessary. No mouse, no separate keyboard, and just a low power processor with a wireless connection to the school network. We can build this now, and run it with an embedded OS to make the unit secure. Manufactured in large quantities with the new flat screen technologies, it will cost less than $200 per unit. If one fails, replacement from spares takes less than 15 minutes and the lesson continues.
The computer technologies are here. But the knowledge delivery systems have yet to appear. I expect this market will see a lot of development in the near future.
Many people employed today are called 'knowledge workers' and it is true in part. Their knowledge is currently built the hard way, with years of experience and the sifting of much information. Not all of the information is current and some of it is probably not accurate, or even true.
None the less, we succeed in this awkward environment because of the ability of the human mind to filter and make sense of diverse and conflicting information. Knowledge systems cannot replace that learning process, but will make organized information available to a wider audience than just employees of companies wise enough to invest in good information systems.
Self employed people, such as myself, are also often knowledge workers. My work is constant research on the Internet interspersed with infrequent trips to the library. I will be an early beneficiary of Digital Libraries, and more so of early KD systems. Information is my beat, knowledge is my product. Knowledge systems will enable me to cover subjects more thoroughly and lead, hopefully, to better writing.
Knowledge systems bring new opportunities to knowledge workers, and new challenges. Once these systems appear, the race to apply them will be on. This will lead to richer opportunities for self employed and company workers. It will also tend to level some of the differences between management and knowledge workers, reducing the current extreme imbalance in power reflected in the layoff figures in 2002.
Companies will be a bit more reluctant to lay off knowledge workers who could instantly hang out a consulting shingle and give generally available knowledge to a competitor. Many freelance individuals will find their specialist knowledge of more value when matched with organized KD.
Is this good or bad for companies? I think KD itself is neutral. It is a tool, not a moral object. How it is used will determine the effect on companies and individuals. My expectation is that most companies will realize the need and implement KD systems quickly. IBM is a leader in this and already has some KD systems in development.
There will be some companies who don't implement KD for various reasons. They can get some support from freelance knowledge consultants, but they will not benefit as much as inhouse KD. However, not all industries are as sensitive to new discoveries as science, medicine and technology. Many will work very well with only outside KD support.
Each change in information technology hastened the delivery of the next change in IT. Mainframes facilitated the design of minicomputers, ditto the minis facilitated micro design, and communications technology prospered in like manner. The big jump came when microcomputers, graphics chips and digital communications added the critical software - a web browser.
Each advance in technology and IT in particular has generated benefits to our economy as companies across the US bought new systems, hired new people and improved their products. Each advance was accompanied by a general expansion of the employment base and increases in the living standard.
This new OSE software will drive an economic expansion longer and further than any before. It will, for the first time, empower us to understand in detail the world we live in and make better choices. This understanding will give us the ability to correct problems and choose our course with knowledge of the consequences.
Since the Internet, changes have happened at express speed. The rest of business has sped up in response to competition and what the Internet enables. Overnight delivery, unheard of by most people ten years ago, is now just a check item from several delivery services. Delivery companies like FedX and UPS have expanded into warehouse management for delivery customers and are now adding inventory management.
Just as Internet shopping has exploded demand for fast delivery services. KD systems and other OSE software will explode the demand for information, filtered, organized and interpreted. A new knowledge industry will spawn rapid advances in science, medicine, technology and education. The personal computer, as we have known it from the late 1970s, will be replaced by diverse information systems where the emphasis is on software delivery of information and knowledge.
The ancient Chinese curse is "May you live in interesting times." They were referring to times of rebellion and war, not technology convergence, but the analogy holds on another level. In war people are displaced and die, in times of rapid technology change, companies are displaced and die, and people are hurt financially in the process.
My view on this matches that of Schumpter's 'Creative Destruction', where the replacement of faltering companies by new ones is the basis for our economy's dynamic strength. In a manner similar to organic evolution, what may be harmful to individual companies is beneficial to the economy as a whole by responding to economic changes with better companies.
KD and OSE will continue the rapid corporate change that hallmarks the Internet revolution. It will be a revolution based on knowledge and software, available to all who want it. This will burden those people and companies whose business is based on our current human information processing. Most will make the transition, but some will not.
In the next ten years, the OSE will create software that will enable things beyond what is written here. But the effects of this software will be very positive for the economy overall. Today's traffic, pollution and education problems can be solved when we can look at these very complex problems and their interactions in detail. KD systems will form the base of an OSE explosion that enables solutions that really solves problems without creating new ones.
Just as information about the earth has enabled us to reduce or eliminate the use of certain chemicals that have very negative long term effects, so will organized information and knowledge enable us to make better choices for everything. This will rapidly spread globally and help solve problems that are beyond our control today.
[30]