Most IT vendor decisions are made in a meeting where the loudest demo wins. The proposal with the best slides looks better than the proposal with the better contract, and a known gap becomes "something we can handle later". A scorecard does not remove judgement, but it forces the judgement into the open.
This guide provides a weighted scorecard you can adapt, scoring anchors so different evaluators apply the same standard, and a worked example. It is designed for an SME buying managed services, software or an implementation partner.
Before you score anything
- Write the requirements. Separate must-haves from should-haves. A vendor that fails a must-have is not rescued by a high average.
- Define the use case. A scorecard for a managed service provider is different from one for a custom software build. Keep the criteria, change the weightings.
- Assign evaluators. Include the person who will operate the service, not only the buyer. Three evaluators with different perspectives is usually enough.
- Agree the evidence. A claim in a proposal is not evidence. Ask for a reference call, a demo scenario, a sample report or a contract clause.
The scorecard
Use a 1-to-5 scale for each criterion, multiply by the weight, then divide the total by five to return to a 100-point scale. The weights below total 100 and are a reasonable starting point for an SME.
- Requirements fit: 20. How much of the must-have scope is covered without custom work?
- Security and privacy: 15. Access control, encryption, logging, incident history, data handling and subcontractors.
- Integration and data portability: 15. APIs, export formats, identity integration and exit support.
- Total cost of ownership: 15. Three-year cost including implementation, licences, support, change requests and internal time.
- Implementation and change management: 10. Plan, team, milestones, training and acceptance criteria.
- Support and service levels: 10. Hours, response times, escalation, credits and named contacts.
- References and track record: 5. Comparable clients, outcomes and willingness to share honest feedback.
- Business stability: 5. Financial health, ownership, key-person dependency and roadmap credibility.
- Contract and exit: 5. Term, termination, price review, data return, transition assistance and liability.
Scoring anchors
Without anchors, one evaluator's 4 is another's 2. Use the same definitions for every vendor.
- 1 — Fails: the requirement is not met, or the answer is evasive.
- 2 — Weak: partially met with significant manual work or an unproven workaround.
- 3 — Acceptable: met with standard configuration, but with a material limitation.
- 4 — Strong: met with evidence, a reference or a clear implementation path.
- 5 — Excellent: met with evidence, tested in a comparable environment, and low ongoing effort.
Score security and data portability strictly. A vendor with a 5 in requirements and a 1 in security should not proceed, regardless of the total.
How to run the scoring session
Score independently before the meeting. Each evaluator records a number and a one-line reason, then the group discusses only the differences of two points or more. This stops the most senior person in the room from setting every score. Keep the evidence beside the number: a reference call, a contract clause or a demo result.
A worked example
Two hypothetical vendors, A and B, respond to the same managed service requirement. The numbers below are illustrative, not real market data.
- Requirements fit: A scores 4, B scores 5. A needs a small custom report; B covers it natively.
- Security: A scores 4, B scores 2. B cannot show a recent independent assessment and stores logs for a short period.
- Integration and portability: A scores 4, B scores 3. A provides a full export; B provides CSV for some modules only.
- Total cost: A scores 3, B scores 5. B is materially cheaper over three years.
- Implementation: A scores 4, B scores 3. A has a named project lead; B's plan is generic.
- Support: A scores 4, B scores 3. A offers business-hours support with a defined escalation path; B offers a ticket queue.
- References: A scores 4, B scores 3. A provides two comparable references; B provides one.
- Stability: A scores 4, B scores 3.
- Contract and exit: A scores 4, B scores 2. B's exit clause requires 120 days' notice and excludes transition support.
Weighted, A scores 77 out of 100 and B scores 70. More importantly, B fails the security threshold and the exit test. The scorecard shows why the cheaper option is not the better one, in a way a steering group can discuss.
Red flags
- A proposal that does not answer the must-have requirements directly.
- No named delivery team, or a team that changes after signing.
- Security answers that rely on a certification logo without a scope statement.
- Pricing that excludes implementation, data migration or support.
- No data export demonstrated with real data.
- References that cannot speak to the specific service you are buying.
- Contract terms that auto-renew with a long notice period.
- Unwillingness to define acceptance criteria or service credits.
Decision rules
- Eliminate any vendor that fails a must-have requirement.
- Eliminate any vendor below the minimum threshold on security and data portability.
- Compare the remaining vendors on weighted total cost, not headline price.
- Prefer the vendor with the stronger implementation plan when scores are close.
- Do not average away a serious weakness in the contract.
- Record the reason for the decision, including the scores and the exceptions. Future you will need it.
After the decision
Use the scorecard as a baseline for the first review. At 90 days, score the vendor again on the same criteria using delivery evidence. If the implementation score drops from 4 to 2, you have a conversation with data rather than impressions. Re-score annually and keep the results with the contract.
Metrics for vendor performance
- Service level attainment against the contract.
- Number and severity of incidents.
- Time to resolve support tickets.
- Change request volume and cost.
- Security findings and remediation time.
- Project milestone adherence.
- Invoice accuracy and cost variance against the three-year model.
Where this fits in our work
If you need help structuring a comparison or reviewing a proposal, start with Operations Systems. The pricing page explains how we scope advisory work. For the budget side of the same decision, read how SME IT budgeting changes year by year.
Sources
- NIST Cybersecurity Framework 2.0, accessed 17 November 2026.
- NIST SP 800-161 Rev. 1, Cybersecurity Supply Chain Risk Management Practices, accessed 17 November 2026.
- ISO/IEC 27001, Information Security Management Systems, accessed 17 November 2026.
- CIS Critical Security Controls, accessed 17 November 2026.
Responses (0)
No comments yet. Add the first one below.
Leave a response
Your email address will not be published. Required fields are marked *.