################################################ # # # ## ## ###### ####### ## ## ## ## ## # # ## ## ## ## ## ### ## ## ## ## # # ## ## ## ## #### ## ## ## ## # # ## ## ###### ###### ## ## ## ## ### # # ## ## ## ## ## #### ## ## ## # # ## ## ## ## ## ## ### ## ## ## # # ####### ###### ####### ## ## ## ## ## # # # ################################################ The following paper was originally presented at the Seventh System Administration Conference (LISA '93) Monterey, California, November 1-5, 1993 It was published by USENIX Association in the Conference Proceedings of the Seventh System Administration Conference For more information about USENIX Association contact: 1. Phone: 510 528-8649 2. FAX: 510 548-5738 3. Email: office@usenix.org 4. WWW URL: https://www.usenix.org Open Systems Formal Evaluation Process Brian William Keves Systems And Network Management Oceanside, CA 92051 Abstract System Administrators and Architects are faced with an abundance of products that can be used to solve a problem or fill a need. Some of these products are not truly compatible with existing or Open Systems hardware and software. They can even be proprietary solutions re-packaged to grab a slice of the "open systems" market. Separating the chaff from the true performers will continue to be an increasingly difficult problem. This difficulty, combined with an increasing trend towards enterprise wide client/server technologies, shows a definite need to qualify procurement techniques and practices. This paper gives administrators the information to successfully organize or re-organize internal policies and procedures to perform appropriate evaluations of Open Systems products. 1 Introduction Most procurement procedures and policies I have seen remind me of the following quote. [ FORMAT: Indent] "At this point, I thought about the four horsemen of application/project development: 1. Conceived in Confusion 2. Born Into Ignorance 3. Developed in Chaos 4. Death by Neglect" [1] [ FORMAT: End Indenting] Although this is not always true, many organizations need help with their entire procurement process. There are many aspects to this beyond simply picking up the phone and ordering equipment. Small organizations usually don't worry about maintenance, compatibility, legal or purchasing issues. Large enterprises must pay attention to these details, since their volume is significantly higher from diverse internal organizations. The problem is that many large organizations still handle the procurement process like a small enterprise. This results in the waste of major amounts of time and money. The main reasons for this attitude are: [ FORMAT: Indent & Bullet] No Requirements No Communication No Standards No Coordination Internal Competition [ FORMAT: End Indenting & Bullets] At some point a large enterprise will look around and see a huge amount of equipment and software that will not work together. Most importantly, the information stored on these diverse platforms is not being shared, leading to unnecessary duplication and lost profit. Every individual and team in an organization will benefit from a planned computing environment. To plan the environment there must be a coordinated procurement process. One of the major portions of procurement is the evaluation process. This paper will go into detail on the formal evaluation process, while still giving an overview of the procurement process and the organizational changes needed. The main goal, as always, is to provide the customer (user) with the best available solution for the lowest cost. 2 Formal Versus Informal Evaluations The difference between the formal and informal evaluation process is mainly in the documentation and scope of the task. Most people use the informal evaluation process, relying mainly on personal judgement and experience to architect a system or solution. The formal process requires proper documentation and an unbiased approach to the evaluation of multiple vendors products. It also brings the customer into the focus of the procurement process. 2.1 Definitions Following are definitions of terms used in this paper. Some of the terms are new and open to discussion, so it is best to be clear on the meanings. 2.1.1 Open Systems This paper uses a market driven definition for Open Systems. If a solution can be obtained from five independent sources or is unique to one vendor but is demonstratively compatible, then it is considered "Open". 2.1.2 Downsizing In conjunction with the trend to reduce the size of a company's workforce and occupied space, downsizing is used to describe the conversion from centralized to distributed computing resources. These tasks are occuring hand in hand and this paper uses downsizing to indicate the trend towards distributed client/server environments. 2.1.3 Customer This is a generic term applicable to all enterprises, including commercial, educational and research organizations. It mainly refers to the user of computing resources that a support organization must architect and administer. 2.1.4 Profit Like the definition for customer, profit is being used to indicate success in an endeavour, however success is indicated in the enterprise. 2.2 Standards Requirements It is very important to understand what standards need to be employed within the enterprise. Europe is dedicated to using formal standards for all products and services. ISO is the main example of this. In the United States there are many formal and informal standards. An example of an informal standard is Sun's NFS. It is an industry or defacto standard that was not developed by an official standards body. [2] The choice of standards is important to the successful conclusion of the procurement process. Research needs to be done concerning what standards an enterprise must adhere to. 2.3 Methodology And Test Standards There are a number of methodologies available which will help with the procurement and formal evaluation processes. Some standards bodies are also writing formal specifications. [FORMAT: Indent and bullet] COS User Requirements Process [3] FURPS [4] IEEE P1003.0 [5] IEEE 1003.3-1990 [6] ISO Technical Report 10000-1 [7] SILC [8] X/Open Open Systems Directive [9] [FORMAT: End Indent and bullet] 3 Why Perform Formal Evaluations? As Administrators are increasingly thrust into the position of architecting and purchasing recommendation and authority, they realize that they are required to provide more justification for a large purchase than "it is the right product to buy." Management wants assurances that the products they are being asked to purchase are going to work now and in the future. "Re-tooling" on the workstation level is much more expensive in a large enterprise than the old centralized approach. For example, a change to an operating system can take days to propogate throughout the network. 3.1 Financial A competitive edge translates into profit. Companies have an obvious bottom line motive, but even educational and research institutes employ the concept of profit. Profit comes from the ability to succeed better than the competition, whoever it is. Information is becoming one of the most important aspects of life today. Organizations that can manipulate and use its information faster and more accurately than others will be the most profitable. Organizations that have forgotten the profit motive and let politics dictate their technical decisions are finding themselves left behind. Downsizing is more than just a trend. It is a direct result of large companies ignoring their infrastructure and finding they can no longer do business. 3.2 Legal This is a subject that many individuals in our field never think will affect them, but they may be in for a surprise. Many large companies, universities and research institutes work on government projects. The requirements for these projects are usually more stringent than in industry, as are the consequences of failure. Sometimes joint partnerships or agreements can also lead to legal complications. When projects fail, no matter who was to blame, reputations can be damaged, companies can go out of business and grants can be lost. It is not necessary to dwell on this aspect, but a formal evaluation process will allow you to show a paper trail of your activities. This documentation is useful for other things, like keeping management informed and interested. It is not unknown for legal disputes to take years to resolve. It is better to have written proof of activities than to rely on memory. 3.3 Professional Our profession is no longer a part time afterthought. We are directly responsible for the productivity of a large amount of users. Their main goal is to perform their work. We have a responsibility to those users to maintain the highest levels of competence and professionalism. Enterprises are growing exponentially and we have to keep up. 4 What To Do First Organizations need to become aware of the need and the money savings of proper procedures. This can be achieved through a two pronged approach. First, put the policies and procedures in place for a subset of the total enterprise. The final step is to publicise the success of the new approach, so other organizations will wish to join in. 4.1 Policy Obviously, management buy-in is critical to set and enforce policy changes, but the user community, the real customer, must not be ignored during this phase. They should help with setting policy, just as they should help with establishing standards and specifying requirements. The earlier that policy is introduced into the life cycle of an enterprise, the faster the results will be apparent. Ultimately, a new venture should have all procurement and administrative policies in place before a single piece of equipment is evaluated and purchased. 4.2 Procedures Flexibility is the key! The goal of procedures and policy is not to create unnecessary red tape, but simply to make an organization smarter in the way they deal with the computing environment. While enterprise wide policy is being established, the procedures necessary to operate can be put in place for all aspects of the procurement process, at least on an interim basis. Many times these "interim procedures" can very quickly become permanent, so it is wise to make the procedures easy to follow and complete from the start. The first step is to get the customer involved with the procurement cycle by encouraging them to plan for future requirements. Since a formal evaluation cycle can take 3 to 6 months, it is important to mold customer attitudes and expectations. It can actually by detrimental to present the support organization as a source of immediate wizardry. This falls apart when enterprises grow beyond a certain point. Next, research a formal evaluation process that is comfortable and design the applicable procedures for the enterprise. All formal standards and known requirements should be included and frequently reviewed for updating procedures. Completion of the process include establishing liaisons and procedures for dealing with Legal and Purchasing. The goal here is to speed the evaluation contract and purchasing processing. 5 Formal Evaluation Process This is a description of a strict, formal evaluation process. This should not be conceived as an immediately implementable goal, but as the desired end result of the process of change within the enterprise. 5.1 Requirements As mentioned above, users and management need to be involved in future requirements planning. A yearly planning cycle seems to work best in today's high-tech enterprises. This does not mean that support organizations should ignore requirements and needs that occur out of cycle. Procedures and budgets should be established with the expectation that emergencies and changes to requirements will occur. If the policy and procedures work, then support organizations will have the time to implement the unexpected. This is preferable to the resource intensive, short cycle, non-communicative planning that seems to be the norm. 5.2 Request For Proposals Whichever the preferred name, Request For Proposals (RFP) or Request For Quotes (RFQ), this document is the corner stone of the evaluation process. While the procurement process can skip a formal evaluation if needed, a formal evaluation cannot exclude a request to multiple vendors. The RFP is absolutely the most important document of an evaluation. This is where requirements and the local environment, as well as existing standards, are communicated to a number of vendors who can satisfy the request. It is a good idea to communicate the RFP widely, as this will provide a large base from which to choose a small number of evaluation units. A complete RFP should consist of the following sections: [ FORMAT: Indented and Bulleted] Purpose Terminology Business Requirements Functional Requirements [2] [ FORMAT: End Indented and Bulleted] An important point to mention is to make sure that the RFP reflects the organization's and user's requirements exactly. Do not "pad" the RFP with irrelevant standards and requirements, since this can make it difficult to succeed in finding the right product for the task. 5.3 Research It is important to research the various aspects of an evaluation. Venders need to be found, products reviews obtained and colleagues queried. Many sources of information are available, in fact too many to read constantly: [ FORMAT: Indented and Bulleted] Trade Magazines Product Directories & Guides Internet - News, Mailing Lists and Databases [ FORMAT: End Indented and Bulleted] 5.4 User Group At the same time the RFP is being written and research is taking place, a small group of users should be assembled on a regular basis to do the following: [ FORMAT: Indented and Bulleted] Help Define Requirements Suggest RFP Recipients Review RFP Responses Form Evaluation Teams Recommend Final Product [ FORMAT: End Indented and Bulleted] When properly organized the users will do most of the work for the entire evaluation. It is the responsibility of the support organization to facilitate meetings and properly document the results as well as set and apply policy and procedures. Properly documented, the results of the user group's participation will provide most of the information necessary to ensure management buy-in. It also encourages users to attempt to solve problems in the organization instead of constantly pointing the finger of responsibility. 5.5 Paper Evaluation Once a suitable number of responses to the RFP have arrived, a paper evaluation is needed. This is the preliminary cut based on the stated capabilities of the product. This is where the research previously mentioned should be used to ensure that the stated capabilities are accurate. This will save the time of doing a physical evaluation on a product that has a known problem or does not really satisfy the requirements. 5.6 Physical Evaluation/Testing Once the list of potential vendors has been reduced to a manageable number, arrangements need to be made to obtain an evaluation product from each of the selected vendors. This should be the current version of the product and not a demo or presentation version. The RFP should stipulate that a 60 day evaluation of the product will be necessary, thereby notifying the vendor of the need to be prepared for this eventuality. Testing for conformance to the standards and requirements is upmost. Second, testing for compatibility with existing equipment. The user evaluation teams should be given clear direction concerning the length of time they have for the evaluation and if the product is large enough, split up the evaluation tasks to different individuals or teams. Each individual must give feedback on the evaluations. Use the vendor's technical support to solve problems and answer questions that arise during the evaluation. This is actually an evaluation task that can easily be accomplished. Workarounds and fixes may be needed, but this should not necessarily invalidate the evaluation. Care must be taken to properly configure and maintain the evaluation product. One hint that will save lots of trouble is to ensure that the users do not perform unreproducable production work on an evaluation product. This means restricting the access to the product to the evaluation team and establishing a clear policy against this behavior. Sometimes, customers are eager to use a new product and will compromise the evaluation by insisting that only the one product they are using can be purchased, which invalidates the reason behind the evaluation, which is to find the product that is best for all users. 5.7 Feedback And Consensus Giving feedback to the vendor during the evaluation is an acceptable practice. It helps the vendor to fix problems and re-direct misconceptions about the product. But don't give them information concerning their competition's product. This can be viewed as unprofessional behavior. Make sure that the evaluation teams give the appropriate feedback needed to make a decision on the product. An evaluation form is the easiest way to accomplish this, possible with a reward as an inducement to complete it. The reward will depend on the situation, but this is a well accepted technique that ranges from candy to bonuses. 5.8 Cleaning Up And Documentation The last step in the evaluation process is to make sure that all unpurchased products are returned to the vendor within the time frame of the evaluation. Some vendors will bill for the product once the evaluation period has expired. Collect all documentation on the evaluation into a report. Present this to management and the user group concerned with the evaluation with a summary explaining the reasoning behind the current purchase. Publish a regular newsletter, in printed or electronic form, which keeps the general user up to date on new products and successful evaluations. 6 Benefits Most support organizations are cost centers, not profit centers. Therefore it is important to save money whenever possible. This will allow better utilization of equipment and administration budgets. This can translate directly into bonuses and favorable performance appraisals. Return On Investment (ROI) is also an important concept in business. Making the most of the money spent for Open Systems technology will help determine the long term success of a company. A formal procedure will help to ensure a proper match between requirements, cost and user performance. 7 Summary The need for definitive Open Systems evaluation methods is apparent from the current state of the industry. This paper is an attempt to convince the reader to start applying formal techniques now, before problems occur. Organizations will live and die on their ability to manage the phenomenal growth in information and technological solutions. 8 Acknowledgements I would like to thank Ross Baker for pointing me to industry methodologies and reviewing my outline. I would also like to thank the Usenix reviewers for their input. Last but not least I thank my fellow consultant John Benton for his inspirational and quotable homilies. 9 References [1] J.T Benton. "30 Years With Computers (And Other Narrow Opinions)." Executive Information Development Company, Barrington, Illinois, 1991. [2] K.M Lewis. "Standards-Based Procurement Using POSIX and XPG." Uniforum, 1993. [3] User Alliance for Open Systems (COS). "User Requirements Process." Ed Albriggo, COS. [4] D.L. Casewell, R.B. Grady. "Functionality, Useability, Reliability, Performance and Support Ability Software Metrics: Establishing A Company Wide Program." [5] IEEE P1003.0 "Guide to the POSIX Open Systems Environment." Kevin Lewis, Digital Equipment Corporation. [6] IEEE 1003.3-1990 "Standard POSIX Test Methods." Lisa Granoien, IEEE Computer Society. [7] ISO Technical Report 10000-1 "Framework and Taxonomy of International Standardized Profiles." Clyde Robichaux, AT&T Corp. [8] "Systems Integration Life Cycle Methodology." SLH Systemhouse. Unpublished. [9] X/Open "Open Systems Directive." Author Information Brian Keves runs his own national consulting firm and has had over eight years of experience in architecting, implementing and managing systems and networks based on Open Systems Client/Server standards. Some of Brian's past clients include Boeing, General Atomics, Mead and The University of California San Diego. He is currently architecting and implementing a nationwide Community Health Information Network for Ameritech. Brian can be contacted at Systems And Network Management, P.O. Box 1819, Oceanside, CA 92051 or via e-mail at keves@Sanm.COM.