Armakuni Contact us
← All case studies

Every customer's exam arrived in its own format.
An expert reformatted each one by hand.

A U.S. learning-platform provider had specialists reformatting course material across eight or nine question types before anything could go live, and a database too old for live analytics. AWS funded a $40,000 assessment that mapped exactly what AI could take on and how reviewers would check it. Then an engineering pod moved straight into building it.

Before

Customers sent their own material, each in its own format.

Analytics ran as batch jobs on an old database.

After
80%cut in exam processing time, targeted
  • A data-readiness baseline, a model evaluation, an architecture, a roadmap and a business case, at no cost.

Customers sent their own material, each in its own format

Enterprise and mid-market customers bring their own training materials to the provider's learning management system (LMS): course content and exam questions, in PDF, Word and everything else. Their formatting and question structures rarely matched what the platform needed.

Every upload therefore went to a subject-matter expert first. Eight or nine question types, each in whatever shape the customer had sent, reformatted by hand before a course could launch. It was slow, and each manual step was a chance for an error to reach a learner.

Analytics ran as batch jobs on an old database

Underneath, the platform ran on MySQL 5.7, which couldn't support the live queries a modern LMS needs for learner performance and content effectiveness. Every analytics request was a batch job against data that was already stale. The business wanted to automate course authoring, generate assessments and learn from learner data, but it couldn't build any of that on data it couldn't see.

We checked the content and the systems before choosing a model

Picking a model and generating questions would have been the quick start. We began instead with what would decide whether any of it could work: could the content repositories, the review capacity and the product integration support automation at all? Without consistent tagging, version control and review, no model pipeline could ingest the content reliably.

How we built it

7 decisions, in the order we made them. Each one says what we did and why, with what it did drawn beside it.

  1. The assessment followed the content, from upload to learner
  2. Models chosen by testing on the real tasks
  3. A person reviews before anything reaches the LMS
  4. The build: one JSON format for every question type
  5. Live analytics without a big migration
  6. The rest of the product, brought up to date
  7. A pod that changes shape with the work

The assessment followed the content, from upload to learner

Upload to learner, examined Metadata1Completeness2Tagging3Versions4 the target: 90% relevance
The engineering

We assessed the whole path from uploaded material to reviewed LMS content, in a three-to-four-week assessment. It mapped author and learner workflows through two to three workshops, and examined text, video and assessment materials along with guidelines, learning objectives and existing annotations:

  • Repository reviews checked metadata consistency, content completeness and access for ingestion.
  • The team identified gaps in tagging, version control and feedback collection that would affect model inputs and the ability to review changes.
  • Content relevance of at least 90% and test pass-rate alignment within ±5% were set as acceptance targets for the evaluation.
  • Proprietary course materials and learner data set the encryption, access and audit requirements for the proposed integration.

Why: we wrote the acceptance target before building anything, so the result couldn't quietly redefine success.

Models chosen by testing on the real tasks

Built withBedrock
The model Candidate modelAnother candidate chosen by testing, changeable
The engineering

Choosing a model on its reputation could mean re-architecting later, so candidate foundation models on Amazon Bedrock were evaluated against the actual content-generation and test-authoring tasks, for relevance, output quality, error types and estimated cost.

The assessment's architecture decision was flexible model selection: the application can change its model as content needs change, balancing cost and performance, without rebuilding the pipeline. The content varies too much for any one model to stay the right choice for long.

A person reviews before anything reaches the LMS

Built withLambdaECS
plannedThe exam ingestedFormattedA person reviews itThen the LMS
The engineering

The planned pipeline connects document intake, question standardization, model-generated content and human review before material enters the LMS:

  • AWS Lambda was selected for exam ingestion and serverless processing, with Amazon ECS for the application services around the LLM pipeline.
  • The integration design connects the LMS, a manual review portal and feedback analytics.
  • Human review stays as the fallback for content that needs judgment or correction.

The build: one JSON format for every question type

Built withBedrockClaude 3.5SQSS3
Question type 1Question type 2… eight or nine One JSON
The engineering

Exam questions had to fit one JSON schema, and customers sent everything but that. The pod's build of the formatting engine runs on Amazon Bedrock with Claude 3.5:

  • Amazon SQS manages multi-document uploads, so a customer's batch is processed reliably, one document at a time.
  • Amazon S3 holds the uploaded documents going in and the formatted JSON coming out.
  • The engine processes raw question files across all eight or nine types and produces standardized JSON the LMS can import.

It's designed to cut exam content processing time by 80%, with every question in a consistent format.

Live analytics without a big migration

Built withMySQLRedshiftNode.jsECS
MySQL 5.7 to 8Zero-ETL to Redshift13 client dashboards
The engineering

The obvious fix for stale analytics is a full data migration. The assessment found a cheaper route. Zero-ETL pipelines can feed analytics close to live, without reworking the data infrastructure, once the database supports them. That was a separate decision from generating course content, and it led to its own work:

  • MySQL 5.7 upgraded to MySQL 8.x, for modern ETL support.
  • Zero-ETL pipelines syncing the upgraded database with Amazon Redshift in near real time.
  • 13 client-facing dashboards covering the LMS's KPI metrics, served by a Node.js service on Amazon ECS, with a dashboard editor so each client controls layout, visibility and component arrangement.

The rest of the product, brought up to date

Built withAngularEC2
The rest, brought up to dateAngular 19Bootstrap 5
The engineering

Alongside the AI and data work, the pod updated the LMS frontend from Angular 13 and Bootstrap 4 to Angular 19 and Bootstrap 5. It also replaced third-party e-signature tools with an open-source e-signature service hosted on Amazon EC2, integrated into the LMS, so the provider owns and controls its own signing.

A pod that changes shape with the work

Built withCDK
One pod CDK, CloudFormationNetworking, CI/CDDevSecOpsData and AI 250 hours a month
The engineering

The assessment moved straight into a flexible engineering pod of 250 engineering hours a month. A lead DevOps engineer supplies 160 of them, with application, data, ML and architecture skills available in the rest. The pod's first scope covered:

  • AWS CDK and CloudFormation for repeatable infrastructure changes.
  • AWS networking and CI/CD work for the application delivery path.
  • DevSecOps tooling built into deployment from the start, so security controls are in the pipeline, not added afterwards.
  • A skill mix that changes as priorities move from platform work into data and AI.

The engineering pod started building as soon as the assessment closed, so its findings went straight into the work.

What we'd tell another CTO

1Assess the content before choosing a model. Tagging, version control and review capacity decided whether any pipeline could work, and the acceptance targets, 90% relevance and pass rates within ±5%, turned "good enough" into a number before the build.
2Look for zero-ETL before planning a database migration; upgrading MySQL to 8.x was a smaller step than moving off it, and it gave live analytics.

What else changed

  • A data-readiness baseline, a model evaluation, an integration architecture, an implementation roadmap and a business case, from a $40,000 assessment that AWS funded, at no cost to the customer.
  • Acceptance targets agreed up front: content relevance of at least 90% and test pass-rate alignment within ±5%.
  • A 250-hour-a-month engineering pod, in place as soon as the assessment closed.
  • MySQL 8.x, zero-ETL to Redshift and 13 client dashboards, plus a modern Angular and Bootstrap frontend and an e-signature service the provider owns.

What comes next

The content pipeline and the analytics deployment in production, with authoring time and learner outcomes measured during delivery. The pod adds data science and ML engineering as the work moves on.

How we'd do it today

This project came before our current offers. Here's how we'd approach the same work now.

The formatting engine is a natural Guaranteed AI Outcome: one AI use case, signed against a number the business already tracks, such as specialist hours per course, live in 90 days or less, with a go-or-stop decision at week 4 and our fee riding on the result. The 90% relevance and ±5% targets would become an eval set built from real uploads, with a sealed second set, and a question that loses its correct answer would be a critical case that has to pass on every run.

Modernize with AI Guaranteed AI Outcomes AK AI Factory

More work like this

  1. AI10–20×A U.S. media archivemore footage tagged per hour of editors' time, in the proof of conceptThe valuable part of an archive film is thirty seconds nobody has found yet.4 days to working AI demos1.2 PB archive planned onto AWSOn AWS S3 · Rekognition · Nova Pro · DynamoDB
  2. AI95%A U.S. support-software companyof shopper questions understood, as reportedShoppers ask with typos, half-sentences and "how long does that take?"90% of answers relevant to the question85% accuracy on intent matchingOn AWS Lambda · S3 · ECS · Fargate
  3. AIMinutesA U.S. smart kitchen brandper recipe draft, where it took hours, in the prototypeA new oven reached customers. Its recipes hadn't.3 tools behind one AI agent2 weeks from first offer to sign-offOn AWS Bedrock · AgentCore · SQS · Transcribe

All case studies →

Have a problem like this one?

A free workshop: a few hours with the architects who'd do the work, on your own systems. No cost.

Book a free workshop