How to run a vendor evaluation that produces a defensible decision

Choosing a software vendor or technology partner is one of the decisions that is hardest to reverse. A contract signed in haste, or on the basis of an impressive demonstration, can bind an organisation to a system that does not fit its actual needs for three to five years. The evaluation process is where that risk is managed, and most organisations underinvest in it.

Requirements before vendors ¶

The evaluation process should begin before any vendor is contacted. The first step is a structured requirements exercise: interviews with the people who will use the system, a review of the process the system will support, and a written document that separates essential requirements from desirable ones.

This document serves two purposes. It gives the organisation a clear picture of what it actually needs, which is often different from what it thinks it needs at the start of the process. And it gives vendors a consistent set of questions to respond to, which makes comparison possible.

Structuring the RFP ¶

A well-structured RFP asks vendors to respond to specific scenarios drawn from the requirements document, not to provide a general product overview. It specifies the format for responses, which makes scoring easier. It includes questions about implementation approach, support model, and pricing structure, not just product features.

The RFP should also ask vendors to describe situations where their product is not a good fit. Vendors who answer that question honestly are generally more trustworthy partners than those who claim universal applicability.

Demonstrations as verification, not discovery ¶

Demonstrations should happen after written responses have been reviewed and scored, not before. By that point, the organisation knows which vendors are credible candidates and has a specific set of scenarios it wants to see the product handle.

A structured demonstration agenda, shared with the vendor in advance, produces more useful information than an open-ended product tour. Ask the vendor to walk through a specific workflow that reflects your actual process. Note where they have to switch to a workaround or acknowledge a limitation.

The goal of a vendor evaluation is not to find the most impressive product. It is to find the product that fits the specific situation, with a vendor that can be held to clear commitments.