Designing Systems to Handle Chaos
Information can be required in a chaotic manner. Assuming that it is somewhere in the computer systems, ordinary office workers should be able to pull it out organized and summarized in a usable form.
  • Requests for information can occur at random
  • Data correlation may be unforeseen
  • Presentation mediums have expanded from old fashioned paper to web pages, PDF files, spreadsheets and word documents.
  • Users should not be required to know data structures
  • Users certainly should not be required to understand queries
  • Complex data relationships and analyses should be easy to extract
  • Filters are common, Options that change the output structure are not
  • Common Wizards do not solve the problem
GUI interfaces need to be developed to make it easier for the ordinary office worker. The article below addresses this problem.

Designing Systems To Handle Chaos (updated to 2003)

Things still haven't changed much.  People now try to download massive amounts of data from a web connection into Excel and manipulate it there.  They have many of the same problems and it is very easy to get wrong answers if they build their own database queries.  This is solved by using a hybrid architect - i.e. a combination of a professional developer and an Options interface for the user, as referenced in my articles in 1982, 1990 and 1997. 

    The basis of Options

This was easy in 1980, but is not with most tools today.  However, WebFocus at www.ibi.com is the child of the product that could do it in 1980 and can easily do it now. See the latest versionat Agile Methodologies and Chaos Revisited (Technical form).

Designing Systems To Handle Chaos (updated to 1997)

The World Wide Web, with its massive amounts of data and search engines, has in many ways increased the chaos in the world. As long as you are searching for one item and there are not too many pages (like 50,000) referencing it, the Web works fine. However, once you need to find something based on summarized data or intelligently linked information, the Web fails.

The original version of this article was published in the Journal of Systems Management in April of 1990. Within the last year, I have seen developers, using the most up to date software, LAN tools and GUI interfaces, doing exactly the same things that programmers did in the 1960's. The tools are better, the GUI interfaces are easy to use, three tier architecture makes more data available, standard wizards are supposed to help the end user, but intelligently analyzed data is still hard to get. It makes no difference where the data is or what tools are being used, the typical user still needs the techniques described below in the original article. Its just that some tools make this easier than others. New comments are bold and italics.


Designing Systems To Handle Chaos
By William Myers, CSP

Tom Peters, in his book Thriving on Chaos, states that "Quality and flexibility will be the hallmark of the successful economy for the foreseeable future." To keep up, computer systems must be just as dependable and flexible. Some computer professionals have long recognized that systems must be flexible. However, a large group is still designing and maintaining systems based on the concepts of the 1960s.

FLEXIBILITY is the key!

Computers have evolved tremendously since the 1960s. Laptop computers can be purchased now that will process the same amount of data that a room full of equipment would process at the time. Software has evolved almost as fast. Unfortunately, the methods used by computer professionals have not always kept up. New computerized information processing systems are still being designed using methodologies, although improved, of the '60s and '70s. These methods are based on the following invalid premise: that the user knows, or the analyst can determine, what is needed from the system. People do not know because things change constantly.

A methodology implies, by definition, that a state of order exists and that the methodology will help the analyst find it. Extended relational analysis will help find the available data and to show the relationships between fields. Data flow diagrams show the processes. A system life cycle controls the design and production of the system. Prototyping gives the user a better feel of a new system. He can try screens and handle real reports. There are other methods, and while these methods all help with things that you know, they are not much use with things that you do not know.

Object modeling combines these paradigms, but still assumes a predetermined structure to the universe.

Data systems have three main functions. First, they serve as a repository for data. Methodologies are best suited for determining what the data is and where it will come from. Second, they may produce a product such as checks or customer information during a phone call. Third, they produce information for analysis. The information required for analysis, and its form, is the most difficult to predict.

The obvious solution is to have the users create their own analysis reports and graphs. The pendulum has swung from having programmers develop all reports to having users develop all reports with fourth generation languages (4GL's). Both are extremes. Not all users want to design their own reports. They may not fully understand the data. I have found that a majority of users do not have the time, aptitude, or inclination to create their own report programs. They would rather have standard report formats that can be altered at tun-time.

GUI Wizards use a better interface and generate code in the background, but still use the same basic concept as earlier 4GL's.

Standard reports produced according to a schedule have two basic flaws: They are rigid and are difficult to analyze. The most common request, once a system has been put into production, is "I like this report, but need a different sequence, totals, portion of the data, etc." Batch or menu requested reports usually require new programs. A large portion of the data processing staff spends time writing these programs and charging time to maintenance.

The ability to handle this type of request needs to be in the hands of the users themselves. Report generators and fourth generation languages can handle this task easily. Standard report programs can be written that expect the report heading, sort sequence, total breaks, and data selection statements as input at the time the program is run. One program can do the job of thousands of rigid programs. The on-line input is simple for the user, but the program itself can be very complex.

I use one today with optional report headings and many data selections that was developed with a popular Microsoft product.

Front end screens can be set up to handle the sorts and selections from the user. The screen will even accept sort (BY) and select (IF, WHERE) statements and insert them in the programs at run time. Imagine sending valid 'if' statements to a COBOL program at run-time. I rarely need to create a new report program once a system is in production.

This is the same as having a program set a property on an object when the user makes a selection. Also, I hear that COBOL is making a comeback

These front-end screens have proven to be extremely popular with the users. In 1982, a manager asked his secretary to call me and request that a new report program be written. She was pleased to tell him that she could create report herself without getting a programmer involved. More recently, a department changed its operating procedures for entering budgets when front-end selection screens were added to its system. Before, one operator had to wait for everyone to enter data before the reports were run. If the operator was gone, everyone waited. Now, everyone runs his own portion of the budget when he has entered his data. In addition, the system has been used to answer top management requests that would have been generated by hand before. The department manager stated that "they use front-end selection screens more than 70% of the time. They don't know how they functioned without them." Monthly reports are the only ones that are run in batch mode.

Without options, I would have to create hundreds of reports to do my current job - I have 6 reports that do everything.

Front end screens have a positive impact on the data processing department. Development, maintenance and storage costs are lower since there are fewer programs. The above budget system went from 77 programs to 22 programs. It would have taken thousands of programs without front end screens to match its power. Few report programs release analysts and programmers so that they can spend more time on tasks that only they can do. However, front-end screens do make programs slightly more complex. The budget system just described required an additional man-day to add these screens.

Many individuals in data processing oppose on-line report requests. They say that the users cannot be trusted to conserve machine resources, can't understand the procedures, or will make numerous errors. These same arguments were used for the last 15 years against on-line data entry. Data entry has moved from a data entry or key punch group to the users, and report generation will do the same. Users are demanding on-line report menus with front-end screens when they see the concept demonstrated.

Those individuals are no longer in data processing, or they have changed their ways.

All systems should have the capability to help the casual user. It's unfair to them to expect everyone to create his own report programs. Reports for new systems should be designed with front-end selection options. Front-end sorts should be included when logical. In addition, there is a tremendous amount of data in existing systems. These systems needed to have reports set up with front-end options. This would add needed flexibility without rewriting, the entire system. When all systems have this flexibility, managers will be able to make better, faster decisions that will allow their companies to compete in the chaos of the 1990's.
Return to Home Page

1997-2015 William W. Myers All rights reserved.
Please direct comments or questions to:  myersww@att.net