Automated testing is usually bought as reassurance. The proposal arrives with a tool name, a coverage percentage and a monthly retainer, and the buyer has to decide whether any of it reduces the risk of a broken release. The useful question is narrower: which failures would hurt the business, and what evidence would catch them before customers do?
This guide is for a company commissioning custom software, replacing a fragile system or taking over a product that already has tests. It covers how to buy testing as a service, how to score providers and what to demand at handover. It is not a guide to repairing a suite that is already failing; that is a separate diagnostic job.
Start with failure cost and release cadence
Two facts shape the purchase. The first is the cost of a defect reaching production: a wrong invoice, an order that cannot be placed, a report that misstates stock or a payment that posts twice. The second is how often the team releases. A monthly release can tolerate a slower, broader test run. A team deploying several times a week needs fast unit and integration checks that run on every change, with a small end-to-end set reserved for critical journeys.
Write both facts down before you speak to a provider.
What the pyramid actually buys
The test pyramid is a portfolio rule, not a law. Unit tests are fast, cheap to run and close to the code; they are good at logic, calculations and edge cases. Integration tests check that components work with a database, queue, payment sandbox or third-party API. End-to-end tests exercise a real user journey through the deployed system; they catch integration and configuration failures but are slower and more fragile.
A healthy portfolio has many unit tests, fewer integration tests and a small number of end-to-end journeys that represent revenue or regulatory risk. The common anti-pattern is an inverted pyramid: a large, slow end-to-end suite, few unit tests and a team that reruns failures until they pass. That suite looks impressive in a demo and becomes a tax on every release.
Accessibility, security verification and performance testing are separate purchases with their own criteria; do not assume a functional test suite covers them.
Provider categories
In-house engineers write tests with each feature
The team that builds the feature writes the tests and owns them. This produces the closest fit between code and checks, and the fastest feedback loop. It requires engineering time, a review standard and a test environment that is reliable.
Delivery partner includes automation in the build
The software partner delivers features with tests and hands both over. This is the normal expectation for a custom build, but the contract must say what "tests" means: which layers, which journeys, what evidence and what happens after handover. Without that definition, testing becomes the first thing removed when the schedule tightens.
Specialist test-automation partner
A specialist joins an existing team to build or rescue a suite, improve CI pipelines or establish a regression pack. This works when the application exists and the gap is testing capability rather than product engineering. The risk is a parallel team that owns the tests while the product team owns the code; the two drift apart unless the specialist works inside the same repository and review process.
Managed device and browser cloud
A device cloud provides browsers, phones and operating systems for running tests. It is infrastructure, not a testing service. You still need tests, maintenance and someone to interpret failures. It is worth buying when cross-browser or device coverage is a real requirement, not as a substitute for a test strategy.
Compare the models on ownership and exit
| Criterion | In-house | Delivery partner | Specialist | Device cloud |
|---|---|---|---|---|
| Test ownership | Product team | Shared, defined in contract | Specialist, then handover | You |
| Ramp-up | Slowest | Built into delivery | Fast on a known stack | Immediate |
| Coverage depth | High with discipline | Depends on scope | High for the brief | None by itself |
| Exit risk | Low | Medium | High without handover | Low |
Evaluation scorecard
Score each provider from 0 to 3 on the criteria, with evidence beside every score. Weight risk coverage, maintainability and handover most heavily; a suite that cannot be maintained is a one-off expense that keeps charging.
- Risk coverage: mapping from business failure modes to specific tests.
- Maintainability: page objects or helpers, readable names, no duplicated setup.
- CI fit: tests run on every pull request, with clear failure output.
- Evidence quality: traces, screenshots, videos and logs attached to failures.
- Test data: synthetic data, reset strategy and no production personal data.
- Handover: documentation, pairing and ownership transfer.
- Commercial clarity: what is fixed, what is time-based and what counts as a change.
The paid pilot
Give two or three providers the same small brief: automate three critical journeys and one integration check against a staging environment. Require the work in your repository, a CI run, a failing test demonstrated on purpose and a short handover note. Then assess whether the tests are readable, whether they run from a clean checkout and whether the provider explains failures without blaming the environment.
A pilot should also reveal how the provider handles flakiness. Ask them to run the suite repeatedly and report the failures honestly. A provider who claims a perfect pass rate on a first attempt is either lucky or not measuring.
Contract terms that decide the outcome
- Who owns the test code, fixtures and CI configuration.
- Which environments and test data the provider may use.
- How failed tests are triaged and within what response time.
- What maintenance is included as the application changes.
- How knowledge transfer happens, and what evidence proves it.
- What happens on exit: repository access, removal of credentials, final report.
Red flags
- A price per test case, with no reference to risk or maintenance.
- An end-to-end-only suite presented as comprehensive coverage.
- Tests that run only on a provider's laptop or a shared environment nobody controls.
- Coverage percentage used as the only quality metric.
- No test data plan, or a plan that copies production customer records.
- No handover until the final invoice is paid.
Choose the model that fits
- Choose in-house when the product will be maintained by your team and release speed matters.
- Choose a delivery partner when you are commissioning a new build and can specify the test evidence in the contract.
- Choose a specialist when a suite exists but is slow, flaky or concentrated on the wrong layer, and you need capability transfer.
- Choose a device cloud only when real device coverage is a requirement and you already own the tests.
If you need a build partner who treats testing as part of delivery rather than an optional extra, start with Custom Software & Integrations. The pricing page explains how discovery, build, testing and support are scoped. For the failure patterns to watch after handover, read automated testing pyramid problems and fixes.
Sources
- Martin Fowler, The Practical Test Pyramid, accessed 17 November 2026.
- Google Testing Blog, Just Say No to More End-to-End Tests, accessed 17 November 2026.
- Playwright documentation, Best Practices, accessed 17 November 2026.
- OWASP Application Security Verification Standard, accessed 17 November 2026.
Responses (0)
No comments yet. Add the first one below.
Leave a response
Your email address will not be published. Required fields are marked *.