Armakuni Contact us
← All case studies

Shoppers ask with typos, half-sentences
and "how long does that take?"

A U.S. support-software company wanted an AI agent that understands shoppers the way they actually write, and answers from each merchant's own policies rather than from a model's guess. We built how it understands a question and how it finds the right policy, added a feedback loop that measures every answer, and made the runtime hold up through Black Friday.

Before

Questions never arrive the way the FAQ is written.

A wrong answer costs the merchant money.

After
95%of shopper questions understood
90%of answers relevant to the question
99.9%uptime through peak shopping traffic

As the company reported.

Questions never arrive the way the FAQ is written

The company serves eCommerce merchants whose busiest shopping days also bring their biggest support queues. Shoppers ask about returns, policies and products in their own words: with a typo, a regional phrase, a half-finished sentence, or a follow-up that only makes sense after the last answer.

Keyword-based support tools couldn't cope. A rephrased question matched the wrong FAQ entry or none at all, and the shopper ended up in a queue for an answer the knowledge base already had.

A wrong answer costs the merchant money

Promotions change the answers while shoppers are still asking questions. A return window, a shipping cutoff, a loyalty rule, if the AI agent gets one wrong, the merchant pays for it through a refund, a chargeback or a lost customer. A model with no grounding will invent those details and sound sure of itself, so the AI agent needed fresh merchant content and enough conversation history to read the next question correctly.

The AI agent understands what shoppers mean and answers from each merchant's own content

The design had two halves. The AI agent understood the shopper by what they meant rather than the words they typed, then answered only from that merchant's own sources, retrieved at the moment of the question, because keeping those two jobs separate is what the rest of the system depended on.

How we built it

4 decisions, in the order we made them. Each one says what we did and why, with what it did drawn beside it.

  1. Matching questions by meaning
  2. Answers grounded in the merchant's own sources
  3. The system scores every answer and tracks results for each merchant separately
  4. A runtime sized for the busiest day

Matching questions by meaning

Built withBedrock
A paraphraseHalf a sentenceA follow-up One intent
The engineering

Two questions that mean the same thing rarely use the same words, which is why the understanding layer works on meaning rather than on the text itself.

  • Amazon Bedrock supplies the foundation models that interpret queries and generate responses, and models can be upgraded as new ones ship without rebuilding the pipeline.
  • Amazon Titan embeddings match by meaning, so paraphrases land on the right intent instead of depending on keyword overlap.
  • Multi-turn conversation handling keeps context across exchanges, so "how long does that take?" refers back to the return being discussed.

Answers grounded in the merchant's own sources

Built withKendraOpenSearchLambda
Kendra indexes FAQs and policiesOpenSearch retrieves the sourceThe answer is written from it
The engineering

Each merchant's knowledge sits in different places, updated on different schedules. We brought it into one retrieval layer, and the model generates only from what it retrieves:

  • Amazon Kendra pulls FAQs, policy documents and product catalogs into the searchable knowledge layer.
  • Amazon OpenSearch handles vector retrieval of the relevant source material, which the retrieval-augmented generation (RAG) pipeline passes into generation.
  • AWS Lambda orchestrates retrieval, generation and integration logic.
  • Index and embedding work targeted sub-second retrieval during peak query volume. The target is for retrieval, not including response generation.

The ingestion pipeline lets authorized merchant updates enter the knowledge layer without an application redeployment, and they reach the AI agent within minutes. That matters most for the policies and product details that change during a promotion.

Why: the model handles language well but doesn't know facts it hasn't been given, so retrieval feeds it those facts and the model turns them into an answer.

The system scores every answer and tracks results for each merchant separately

Built withSageMaker
Shopper ratingsStaff correctionsEscalations Per merchant
The engineering

An AI agent that can't learn will still get a question wrong on day 90 if it got it wrong on day 1. We capture feedback on each response so it doesn't stay buried in the conversation history.

  • Shopper ratings, support-staff corrections and escalation outcomes are stored with the interaction's context.
  • An Amazon SageMaker pipeline aggregates that feedback, evaluates model performance and surfaces candidates for refinement.
  • Reporting separates accuracy and resolution by merchant, so the team can investigate performance against a particular catalog and knowledge base, and show each merchant how the AI agent does on its own customers.

A runtime sized for the busiest day

Built withECSFargateRDSS3CloudWatchIAM
sized for the busiest day Shopper questionsRuntime
The engineering

Black Friday and merchant promotions multiply support volume within hours, and an AI agent that slows down at peak causes real problems for shoppers trying to get help. The runtime uses containers on Amazon ECS/Fargate with autoscaling so capacity follows demand. Amazon RDS holds transactional state with merchant-level isolation, which means one merchant's shoppers and conversations can never reach another's. Amazon S3 stores conversation history, feedback and audit data, with lifecycle policies for retention.

Encryption and access controls protect the stored data and the service interfaces, CloudWatch tracks latency and availability, IAM sets the access boundaries and AWS WAF protects the public traffic path. Because security is built in this way, a merchant's security review gets its answers straight from the architecture.

What we'd tell another CTO

1Understanding the question and answering it stay two separate jobs. Meaning-based matching finds what the shopper is asking, and retrieval from that merchant's current content supplies the answer, so the model itself is never the source of a return policy.
2A content update is a data change rather than a deploy, because promotions move faster than release cycles, and the ingestion path gets merchant changes into the AI agent within minutes.
3Report accuracy per merchant, because each merchant cares about its own catalog rather than the platform average.

What else changed

As the company reported.

  • 85% accuracy on semantic intent matching across paraphrased queries.
  • A learning loop in production, with per-merchant accuracy reporting.
  • Merchant isolation and security built into the architecture.

What comes next

Next, AI agents take on the work itself, handling returns, exchanges and order changes without a person stepping in, and we're extending support to merchant verticals beyond core eCommerce.

How we'd do it today

This project came before our current offers. Here's how we'd approach the same work now.

The next step, acting on returns and order changes, is where AK AI Factory's controls matter. Each tool into the merchant's order system (look up, check eligibility, issue, confirm) would get its own permission and a plan for when it fails, each AI agent its own identity in AgentCore Identity with Cedar policies at the gateway, deny by default and checked on every tool call, and everything a shopper types would be treated as untrusted input. A golden set of real past conversations per merchant would gate every change, with a wrong refund counted as a critical case that has to pass on every run.

Have a problem like this one?

A free workshop: a few hours with the architects who'd do the work, on your own systems. No cost.

Book a free workshop