A hundred and fifty people listened to calls all day.
The question was whether a smaller model could do the job.

A model is only as good as what it's given, and this platform's pipelines failed quietly. We mapped the whole data platform first: an Amazon S3 data lake with Raw, Bronze, Silver and Gold layers, the Apache Airflow DAGs that moved data through it, the adapter framework and the third-party integrations. That gave the team a documented current state for the first time, with each place it failed.

Every data source had its own custom adapter, so each new one meant building integration logic again. The highest-volume call integration was also the most failure-prone, so we rebuilt it first, as the proof of concept for one standardized adapter:

• Data-quality validation on every incoming record.
• Automated error handling and retry logic for failed processing.
• Structured logs that put failures in front of engineers before they reached clients.
• A reusable pattern for every adapter after it.

Around it, AWS Glue took on automated cataloging and data processing, AWS Lambda handled lightweight serverless computation, Amazon CloudWatch added monitoring and alerts across the ETL pipelines, and infrastructure-as-code templates made environments reproducible, so nobody had to rebuild a deployment by hand. If the labels feeding an evaluation are wrong, the evaluation is wrong too, so the pipeline had to be trustworthy first.

Public benchmarks don't tell you whether a model can label this business's calls. So we built the test from its own work: 300–500 human-validated, labeled calls, scored with the customer's existing evaluation framework.

• Claude Sonnet was the existing call-analysis model, and the accuracy baseline for classification, disposition labeling and outcome extraction.
• Amazon Nova was the smaller option, for call and form analysis. Out of the box, the base model didn't match Sonnet on these tasks, so we used Amazon Bedrock fine-tuning workflows to adapt it, on labeled calls, form submissions and the less common service categories where a general model is weakest.

We compared Nova's base and fine-tuned versions against the labeled examples, and broke the results down by prompt and by output value. So the team could see exactly which classifications each model got right or wrong, not just one average.

Subject-matter experts reviewed classifications and refined prompts inside the same evaluation framework, without needing deep ML expertise. That puts the model's quality in the hands of the people who know the calls best.

Why: if only the people who built a fine-tuned model can check it, the business depends on them; when its own experts can test it, the business keeps control.

A model that's as accurate but costs more is no win, and one that's cheaper but slightly worse can still cost the business its clients. So the evaluation weighed task accuracy against operating cost:

• Cost-per-call breakdowns showed the economics of each model option.
• Total cost of ownership covered 3, 6 and 12 months.
• A go/no-go guide brought the accuracy and cost findings together for the decision.

The deliverable was a tested model option, and the evidence to decide on a switch. The savings depend on production adoption and operating results, and we said so.

The prompts were the other bottleneck. In a separate proof of concept, we built one shared prompt structure for calls, forms, schedulers and chats, with modular, dynamic sections that bring in the right context for each engagement type:

• A team member can change one block without rewriting the whole prompt.
• Langfuse keeps the version history, so every change can be traced.
• Side-by-side model comparisons show what a change did to the outputs.
• Tags decide which prompt version is in use, so a change goes live on purpose.

With the foundations in place, we built three more proofs of concept to answer the questions the platform's clients were actually asking.

Which service opportunities were missed? A post-call engine reads each transcript for recurring customer problems, no previous maintenance and signs of dissatisfaction. It maps those signals to a service recommendation library covering maintenance agreements, seasonal tune-ups, installation upgrades and complementary services, then scores each opportunity so the business can see where a relevant recommendation went unmade.

What should clients expect from their marketing spend? A forecasting engine uses historical service-job data, seasonality, campaign type and close-rate history. Time-series forecasts give expected lead volume, qualified leads, revenue ranges and confidence scores. A budget recommendation model weighs return on advertising spend and channel performance, and suggests where to spend more or less.

Which campaigns need attention? An anomaly-detection engine watches call counts, qualified leads, cost per lead, job value and campaign performance. Seasonal adjustment tells an expected dip from a real problem, configurable severity thresholds decide what's worth an alert, and alerts go to the internal team to investigate.

Together with the prompt framework, these were four proofs of concept.

Text transcripts lose the tone of a call: the frustration, the hesitation. Amazon Nova Sonic was assessed for voice-tone and intonation analysis, using a parallel audio-stream design, along with speaker identification: voice-print matching for repeat callers, automatic tagging of the people on calls and speaker verification.

Audio brings biometric obligations, so the assessment covered GDPR and CCPA, encryption and access control for audio streams and anything derived from them, and the audio quality each capability would need. The roadmap puts speaker identification at 0 to 3 months and the full voice-analysis pipeline at 3 to 12 months. This stayed assessment and roadmap work.