There's a particular kind of deploy anxiety you get when your software sits between a provider and getting paid. A bad release doesn't just annoy users — it stalls claims, which stalls revenue, for people who are not your employees and did not choose your release schedule.
Two tools changed how my team ships: Testcontainers for integration tests, and Argo Rollouts for progressive delivery. They solve different halves of the same problem, and the order you adopt them in matters.
The Half That Tests Solve
For years our integration tests ran against mocks or a shared test database. Both are lies, in different ways.
Mocks test that your code calls the API you think it calls. They don't test that the query returns what you think it returns, that the migration applies cleanly, or that your transaction isolation assumptions survive contact with a real engine. A repository test against an in-memory fake proves almost nothing about the repository.
A shared test database is more honest but introduces the worst property a test suite can have: order dependence. One developer's test data leaks into another's assertions, someone runs the suite locally while CI is running, and you get failures that nobody can reproduce. Teams respond to flaky tests by ignoring them, which is worse than not having them.
Testcontainers fixes this by spinning up a real PostgreSQL in Docker, per test class, throwaway.
public class ClaimsRepositoryTests : IAsyncLifetime
{
private readonly PostgreSqlContainer _db = new PostgreSqlBuilder()
.WithImage("postgres:16-alpine")
.Build();
public async Task InitializeAsync()
{
await _db.StartAsync();
await using var ctx = CreateContext(_db.GetConnectionString());
await ctx.Database.MigrateAsync();
}
public Task DisposeAsync() => _db.DisposeAsync().AsTask();
}
That MigrateAsync line is quietly the most valuable part. Every test run applies your migrations from scratch against a real engine. Broken migrations stop being a production discovery.
Real Postgres also means you test the things that only exist in real Postgres: JSONB operators, partial indexes, ON CONFLICT semantics, actual constraint violations. We caught a unique-index assumption that every mock in the suite had been happily confirming for months.
The cost is honest: container startup adds seconds. Reuse the container across a test class, run classes in parallel, and pin the image tag so CI isn't pulling a new digest on a random Tuesday.
The Half That Tests Can't Solve
Here's the thing nobody says loudly enough: a green test suite means your code did what you expected under the conditions you imagined. It says nothing about production traffic, production data volumes, or the payer integration that behaves differently at 400 requests a second than it did at four.
That's the gap progressive delivery fills. Instead of replacing all pods and hoping, you move a slice of real traffic to the new version and measure.
strategy:
canary:
steps:
- setWeight: 10
- pause: { duration: 10m }
- analysis:
templates:
- templateName: error-rate-and-latency
- setWeight: 50
- pause: { duration: 10m }
- analysis:
templates:
- templateName: error-rate-and-latency
- setWeight: 100
The analysis steps are what make this more than a slow deploy. An AnalysisTemplate queries your metrics — error rate, p99 latency, whatever actually indicates health for that service — and fails the rollout automatically if the canary is worse than baseline. No human watching a dashboard at 2am deciding whether 0.4% is bad.
Two things I'd tell anyone setting this up:
Pick metrics that fail fast and mean something. Our first template measured a window so long that a bad canary served errors for twenty minutes before anything tripped. Error rate over a short window catches the obvious breakage; latency catches the subtle kind, which is usually a query plan you didn't expect.
Make rollback boring. The value of automated analysis isn't the automation, it's that reverting stops being a decision someone has to justify. When rollback is a normal event rather than an admission of failure, people ship smaller changes more often — which is the actual goal.
Order Matters
If I were starting over: Testcontainers first, Argo Rollouts second.
Progressive delivery on top of a test suite you don't trust just means you find your bugs in production more gradually. The canary catches what tests can't — load, real data, integration behaviour — but it shouldn't be catching a broken migration or a repository method that was never right. That's expensive detection for cheap bugs.
Get the suite honest first. Then use progressive delivery for the class of failure that only production can reveal.
What Changed
Deployment frequency went up and change failure rate went down, which is the pairing you want — either one alone is easy to fake.
But the change I actually care about is cultural. Deploys stopped being events. There's no Friday freeze, no war room, no deploy champion. Somebody merges, the canary takes ten percent, the metrics hold, it goes to a hundred. Most of the team doesn't watch.
That's the real return on this kind of infrastructure: not speed, but the removal of fear. Engineers who aren't afraid of their deploy pipeline make better decisions about everything else.
