Home > Papers > How to evaluate a content management system

Contact | About | Search

	(772 PDF)       KM Column Content Management Knowledge Management       IA Essentials (Brisbane)             Content Management Requirements Toolkit       Component of The CMS Lifecycle        Interested in syndicating our articles? Find out more JANUARY 2002 How to evaluate a content management system by James Robertson Published on 23 January 2002 Selecting and implementing a content management system (CMS) will be one of the largest IT projects tackled by many organisations. With costs running into the millions of dollars, it is vital that the right CMS package be selected. This article outlines some of the lessons that we have learnt when assisting clients to chose a CMS. It offers ideas and tips, and provides an approach for identifying your business' actual requirements for a CMS. With so many vendors and products, it can be very hard to compare between them. Preparation, and a disciplined approach to this evaluation process is critical. What this article isn't No vendors or products are mentioned in this article: this is not a survey of current commercial solutions. Instead, it provides tools to assist you to conduct a review of suitable products. There is no "one size fits all" solution: no two organisations have the same requirements. Assumptions In developing these guidelines, we have made several key assumptions about the type of organisation purchasing a CMS: medium to large organisation current publishing systems will be replaced by the new CMS CMS will manage both the intranet and internet website CMS will be enterprise-wide. A wide range of content will be published using the CMS. This can be characterised as: simple pages complex pages, with specific layout and presentation dynamic information sourced from databases, etc training materials online manuals (policy & procedures, HR, etc) general business documents thousands of pages in total extensive linking between pages. Ask yourself: what are your business goals and needs? Business goals & strategies Why are you purchasing a CMS? Before identifying specific requirements, you must determine the business goals that will be achieved by implementing a CMS. These must also reflect the long-term strategies and directions of your business. It should be possible to succinctly outline your business goals on a single page. Make sure these are well-understood and agreed to by all stakeholders before starting the requirements gathering process. Identifying requirements There is no single best list of requirements for a content management system. Every organisation has unique needs. Involve all your stakeholders in the requirements process. This includes relevant IT groups, business units, and end users. This is particularly important if you are purchasing an "enterprise-wide" CMS. Use structured investigation methods, to ensure that the list of requirements is both manageable and sufficient. If this process is approached in a disciplined way, there is little danger of the project suffering from "feature creep". Structuring requirements The list of requirements for an enterprise-wide CMS will grow quite large. Group the items into categories, to make this list more manageable. One classification scheme that has worked well for us is: Content creation Content management Publishing Presentation Contract & business This list covers the full lifecycle of a content management system, from initially creating the content, through to delivering it to end users. Ideas box This section gives you some starting points for your requirements gathering process. These ideas have been distilled from the CMS projects we have been involved in. This is far from a complete list, and is no replacement for a full requirements gathering process. (See the Content Management Requirements Toolkit for a comprehensive list of over a hundred individual requirements.) Content creation This is the functionality required by the authors (content creators) using the CMS. Without an effective authoring process, use of the CMS will wither and fail within a year of implementation. Key requirements may include: Integrated authoring environment The CMS must provide a seamless and powerful environment for content creators. This ensures that authors have easy access to the full range of features provided by the CMS. Separation of content and presentation It is not possible to publish to multiple formats without a strict separation of content and presentation. Authoring must be style-based, with all formatting applied during publishing. Multi-user authoring The CMS will have many simultaneous users. Features such as record locking ensure that clashing changes are prevented. Single-sourcing (content re-use) A single page (or even paragraph) will often be used in different contexts, or delivered to different user groups. This is a prerequisite to managing different platforms (intranet, internet) from the same content source. (This is a complex requirement that warrants a whitepaper of its own.) Metadata creation Capturing metadata (creator, subject, keywords, etc) is critical when managing a large content repository. This also includes keyword indexes, subject taxonomies and topic maps. Powerful linking Authors will create many cross-links between pages, and these must be stable against restructuring. Non-technical authoring Authors must not be required to use HTML (or other technical knowledge) when creating pages. Ease of use & efficiency For a CMS to be successful, it must be easy to create and maintain content. Content management The core of most CMS solutions is a central repository, supported by a range of tools for manipulating and managing the content. Key requirements may include: Version control & archiving Strict version control is necessary for legal accountability, backup and disaster recovery. A simple but powerful interface must be provided for these features. Workflow Decentralised content creation relies heavily on a powerful workflow model, that can be easily customised, and is resilient against organisational change. Understand the needs of your authors: they will be doing all the hard work Security Adequate security levels and audit trails must be in place to protect the integrity of the content. Integration with external systems A CMS is typically only one of a number of systems used to present information on the intranet or website. An enterprise-wide CMS will only be successful if it can be cleanly integrated with existing business systems. The mechanisms for achieving this must be fully documented, and based on open or industry standards. Reporting The CMS must provide an extensive range of reports, for both users and administrators. Ideally, the system should pro- actively report on any issues that arise. Support for customised reporting is also desirable. An enterprise-wide CMS will fail unless all stakeholders are involved from the outset Publishing The publishing engine takes the content stored in the repository, and generates the final pages. While this may be a dynamic or batch process, the same basic requirements apply. Key requirements may include: Stylesheets Final appearance is controlled through the use of stylesheets. This provides flexibility and expandability. Page templates Overall page layout is specified via page templates. Ideally, a non-technical interface should be provided for managing this. Extensibility It must be simple to integrate code "snippets" (or equivalent) to provide additional publishing functionality. The CMS must support a process of "continual improvement" in interface design. Support for multiple formats The CMS must publish to multiple formats, such as: HTML (web), printed, PDF, hand-held (WAP), and more. It should be possible to add support for additional formats, which will be necessary as new standards evolve. In order to achieve high-quality in every format, it is critical that the content be separated from presentation at the time of authoring. This allows distinct stylesheets to be used for each output. The most important part of a CMS is the content itself Personalisation Different information is presented based on either user profiles, or metadata in the source content. This is typically required for large "portal" websites. Usage statistics The CMS must allow comprehensive usage statistics to be gathered, including: most popular pages, daily usage, and search engine usage. This information allows the success of the site to be tracked, and any usability issues identified. Presentation The published pages must meet certain standards if they are to be of value to your users. It is important to specify these requirements if you are asking the vendor to design the appearance and layout of the web pages. Key requirements may include: Usability This covers aspects such as ease of use, learnability and efficiency. Usability can be assured by conducting tests on the prototype designs with real users. Usability heuristics (guidelines) must also be followed. Accessibility The CMS must conform to standards such as the W3C Web Accessibility Initiative (WAI). Cross browser support The pages must be viewable in all major web browsers (Internet Explorer, Netscape, Opera, etc). Specify which browser versions are to be supported. Limited client-side functionality You may wish to limit which client-side technologies (Java, JavaScript, Flash, etc) are required to view the site. This is more important for a website than an intranet. Speed Page size must be limited to ensure that load times are acceptable for users. Specify the typical user access methods (LAN, modem, cable, etc). Valid HTML All pages must conform to the current HTML specification. This ensures maximum compatibility across browsers and platforms. Effective navigation Users must be provided with consistent, comprehensive and usable navigation aids. Metadata All pages must provide sufficient metadata to allow effective indexing and searching. This should conform to a standard such as Dublin Core. The total cost of running a CMS includes the skills and resources needed within your business, not just the dollar values Contract & business Project management and business requirements must also be satisfied in a CMS project. Key requirements may include: Training The vendor must list the training materials that exist for the CMS, and the training services that they can provide. Documentation The CMS must be supported by adequate documentation: for users, administrators and developers. Warranty The warranty period provided, once the software has been purchased. Maintenance agreements The vendor must outline their preferred support arrangements, including service level agreements and upgrade processes. Resources required The hardware, software and operating systems required by the CMS. Skills required What skills and knowledge will be required within your organisation to customise and maintain the CMS? Cost Both the fixed costs for the CMS, and the per-user ("per-seat") costs. The latter is generally more significant for a large organisation. Scalability The load levels that the CMS supports, and the additional resources (hardware & software) required for increased usage. IT constraints Specify any pre-existing hardware or software that the CMS must interface with, or run on. This includes specific operating systems, databases or webservers. Reference sites The vendor must supply a number of sites where the software has been successfully implemented. These must match the characteristics of your organisation. Avoid jargon, as there is little common understanding of content management terms, even amongst vendors Some tips When developing your list of requirements, keep these guidelines in mind: Understand your content Conduct a thorough survey of the information that is to be published via the CMS. The nature of this content will influence the functionality required. It is important not to lose sight of the content when selecting the content management system. Relate to business needs Every requirement must be associated with one or more business needs. This allows you to identify the reasons for the requirements, and to prioritise them. Avoid technical details Ensure you specify business needs, not implementation details. Vendors should be free to propose any methods or technologies that are able to meet your goals. Avoid jargon, as there is little common understanding of content management terms, even amongst vendors Provide descriptions Your challenge is to make your needs understood, despite differences in vendor backgrounds and understanding. Avoid using jargon terms, and spell out your requirements in as much detail as possible. Use examples Examples provide concrete situations and business needs, and are an effective way of supporting your requirements. Use wherever possible. More not less It is better to have too many requirements than too few. The CMS must work for your entire business for several years at least: do not be surprised if the final list of requirements is long. Evaluating vendor products Once your requirements have been identified, these must be used to select from the (potentially) large list of vendors. There are a number of approaches that may prove useful: Formal tender The vendors must be required to provide detailed descriptions of how each of your requirements will be met by their system. These responses may be gathered in a variety of ways, but the most common is a formal tender process. Using this approach ensures that the vendors are accountable for any promises or commitments they make regarding their CMS. Demonstrations You must ensure that vendor demonstrations are more than just a sales pitch. To be of value, they must demonstrate how the product will meet your business' needs. The best way to achieve this is to develop scenarios. These describe common or important tasks that will be performed using the CMS. By presenting these in a "narrative" form, considerable scope can be covered in a relatively brief description. Provide these to vendors at least a week before the presentations, and require strict adherence. This allows direct comparison between the vendors' products. Scoring Whatever evaluation processes are followed, you must eventually chose a single successful vendor. To do this in an impartial way, create a scoring system. Determine this before the vendors are contacted, and incorporate the results of any tenders or demonstrations. Using a formal scoring system eliminates the potential for accusations of bias or corruption. Conclusion Selecting an enterprise-wide CMS is often a multi-million-dollar exercise. It is therefore critical that the new system meets your current and projected needs. The single most important activity is to identify your business goals and requirements. This process must involve all relevant stakeholders. Only once you have a full list of requirements are you in a position to compare vendor products. Spending time on these initial phases reduces the business risks inherent in purchasing a CMS. The project is also likely to be more successful (it will do what the users need), and development efforts will be reduced (goals are clear). (For further information regarding the evaluation and selection of a CMS, download the Content Management Requirements Toolkit.) About the author James Robertson is the managing director of Step Two Designs, an intranet and content management consultancy based in Sydney, Australia. James specialises in intranet strategy, web content management, information architecture and usability. If you have any comments on this article, please send them to: jamesr@steptwo.com.au Printed version of this article (PDF)

	For questions or comments please contact the Webmaster. Last updated 19 March 2006. © Copyright 2006, Step Two Designs Pty Ltd, ABN 99 081 019 623. All rights reserved. Intranet Peers and Intranet Roadmap are trademarks of Step Two Designs.