Contact us
A U.S. health-data platform answers questions about drug formularies and insurance coverage, and the same question kept triggering the same expensive model call while large PDF jobs slowed the application for everyone. We put a cache in front of repeated questions and made model choice a configuration setting instead of something buried in code. Heavy PDF processing moved onto its own workers that scale separately, and every processed batch now gets a version so results can be compared later.
Every time someone asked the same question, the system billed for it again.
The same servers ran both the big jobs and the small app.
As the platform reported.
The platform processes drug formulary and insurance-coverage data so providers, insurers and patients can find out whether a medication is covered. Many questions are the same question asked again: does this plan cover this drug? Each one triggered a full model call, at full price, even when the answer hadn't changed since an hour ago.
The PDF jobs that turned new formulary documents into searchable data ran on the same servers as the main application, so the whole system slowed down whenever things got busy. File uploads and downloads went through the application servers too. Testing a different model meant changing code and restarting servers, which made comparing models slow. There was also no way to tell which version of the processed data an answer came from, which made changes hard to compare or audit.
The fix was to give each kind of work its own path instead of pushing everything through the same one. A repeated question now hits a cache instead of the model, since asking it again costs nothing. A heavy document goes to workers that scale on their own, which suits PDF processing's different demands from answering a question. Files move straight to storage, and switching models is now a matter of changing a setting, without redeploying anything.
3 decisions, in the order we made them. Each one says what we did and why, with what it did drawn beside it.
ElastiCache stores earlier answers for reuse in the repeated-question workflow, so a common coverage question comes back from the cache instead of the model. Every processed batch gets its own version ID, and that identifier links each result to the data it came from, so a cached answer is always tied to a known version of the formulary.
Why: in coverage data the answer only changes when the formulary does, so the cache key is the question plus the data version.
A configuration layer lets the team switch models without changing application code or restarting servers. The stack includes Bedrock and Nova Pro, and the architecture supports evaluating alternatives, so a better or cheaper model can be tried in minutes.
The slow, costly jobs and the quick, cheap jobs no longer compete for the same machines, because each one now has its own path.
As the platform reported.
More model evaluation on the configuration layer, and the same versioning applied across the data pipeline.
This project came before our current offers. Here's how we'd approach the same work now.
Most coverage questions are routine, so AI Inference Engineering would build the platform its own model for them on Amazon SageMaker AI, at about a tenth of the price, with a router sending each request to the smallest model that passes the tests and the hard ones to a frontier model through Amazon Bedrock. Before any switch, a golden set of real coverage questions with the right answers, plus a sealed set nobody tunes to, would have to pass, with a wrong coverage answer counted as critical.
Agent ControlPlane, part of AK Run, would make Amazon Bedrock AgentCore Gateway the only way to a model and record each call with the model used, who asked and what it cost.
A free workshop: a few hours with the architects who'd do the work, on your own systems. No cost.
Book a free workshop