An outage nobody could explain for months.
Explained in 60 hours.

A Canadian direct-to-consumer insurer's platform went down every few months, and every restart wiped out the evidence of why. We put two senior engineers onto the last outage's surviving logs, traced the failure to Apache threads held open behind about 800 database queries per request, and set out fixes that release stuck threads without ever interrupting a policy being bound or a payment being taken.

Before

The restart that destroyed the evidence.

The standard fix would have broken the business.

BeforeAfter

After

60 hoursto explain outages that had gone on for 6–10 months

Fixes designed around the atomic transactions: scoped session timeouts and a separate pool.
A plan to close the visibility gap, with RDS Performance Insights and request tracing.

The restart that destroyed the evidence

The insurer sells home and tenant insurance directly to consumers, processing about 1,000 policies and more than 2,000 quotes a day through two production servers on a LAMP stack. Every few months, Apache threads climbed to the 150-per-server ceiling and stopped releasing, and the site started returning 404 and 500 errors.

The team restarted the servers, which fixed it, and erased every trace of why it had happened. For six to ten months, the cause stayed a guess. There was no monitoring on the database or the application, so every incident was investigated blind.

The standard fix would have broken the business

The suspicion was the database: each request generated about 800 MySQL queries, and something between Apache's held threads and MySQL's session behavior was failing. The textbook remedies for thread exhaustion are aggressive timeouts, tighter connection pools and recycling threads. But this platform binds policies and takes payments as atomic transactions. A blunt timeout or pool change could cut one off mid-flight, leaving a customer charged without a policy, or covered without paying.

Read the evidence before it's gone

The turn was simple and nobody had done it: keep the evidence from the next outage, and read it. Then design the fix around the transactions that must never be interrupted, rather than around the generic advice.

How we built it

2 decisions, in the order we made them. Each one says what we did and why, with what it did drawn beside it.

Start from the outage, not a reproduction
Release stuck threads, never a transaction

Start from the outage, not a reproduction

Built withMySQL

The engineering

A 60-hour pod of a senior, MySQL-focused CloudOps engineer and a senior application engineer worked under the insurer's direction, inside its production AWS environment:

We preserved the incident evidence from the last outage before a restart could erase the failure sequence.
We correlated Apache thread lifecycles with MySQL sessions and deadlock behavior.
The working theory was that database sessions weren't releasing, so the analysis went to database-layer contention first and ruled out the candidates a generic playbook tests first, such as application memory leaks or PHP process lifecycle issues.
We examined request-level query traffic alongside the application servers' behavior.
We tested every remediation idea against the atomic transactions for policy binding and payment, and ruled out anything that risked them.

Release stuck threads, never a transaction

Built withRDSMySQL

The engineering

A broad timeout or pool change could interrupt policy binding or payment mid-transaction, so the fixes separate those atomic flows from general traffic:

Session timeouts scoped to release stalled requests before threads pile up at the 150-per-server ceiling.
Separate connection-pool behavior for policy binding and payments, apart from general request traffic.
Amazon RDS Performance Insights on MySQL, to see query performance and session behavior.
Request-level tracing in the application, to connect a blocked call to its database work.

Why: in an insurance platform the cheapest failure is a slow page and the most expensive is a half-finished payment, so the fix had to know the difference.

One stuck thread, end to end

A quote request arrives and starts its roughly 800 MySQL queries.
One session stalls in the database, and the Apache thread waits on it.
Before: threads like it piled up to 150 on each server, the site errored, and a restart erased the evidence.
After: a scoped session timeout releases the stalled general request, and its thread returns to the pool.
A payment in progress runs in its own pool, outside that timeout, and completes.
Performance Insights and request tracing show the stalled query, so the team can fix it at the source.

What we'd tell another CTO

1Preserve the evidence before you restart, because it's the only way to explain an outage you can't reproduce. Then design the fix around the transactions that must never break, since generic tuning advice doesn't know your business, and give the critical flows their own pool and their own rules.

2Put eyes on the database first. At 800 queries a request, that's where the answer was, and it's why the pod paired a MySQL specialist with an application engineer.

What else changed

Fixes designed around the atomic transactions: scoped session timeouts and a separate pool for policy binding and payments.
A plan to close the visibility gap, with RDS Performance Insights at the database and request tracing in the application.
A retained pod: 30 hours up front and 30 held for the insurer to call on, for DevOps, database and application work.

What comes next

Deploying the observability stack and the reliability changes, on the retained pod.

How we'd do it today

This project came before our current offers. Here's how we'd approach the same work now.

Today this would be our managed service, all four parts of it run by one team, so the insurer doesn't staff a night shift. In production, AK Run's AI agents would capture the evidence at the first sign of thread build-up and draft the fix, so a restart never wipes the trail again, with RDS Performance Insights and request tracing on the same watch. The application would be ours to fix as well: a stalled query gets fixed in the PHP code, with a test, as a pull request the insurer's approver ships through its own pipeline. Patches, upgrades and certificates on the LAMP stack would run on a calendar with an owner for each, and any AI agent the insurer adds later would have every model call go through Agent ControlPlane's one gate.

A named service lead would answer for all four, and we'd own each finding until a tested fix is live, with no standing admin access: only roles the insurer grants and can revoke, and a break-glass role its on-call approves. The way in is two weeks of read-only access, in which AK Run replays the last quarter's incidents, alerts, releases and bills, and shows what it would have caught. For the companies we run, the median AWS bill falls 40% in the first quarter.

AI-Native Managed Services Modernize with AI

More work like this

All case studies →

Have a problem like this one?

A free workshop: a few hours with the architects who'd do the work, on your own systems. No cost.

Book a free workshop

An outage nobody could explain for months. Explained in 60 hours.