Armakuni Contact us
← All case studies

Finding one psychological test in a paper
took an expert three hours.

A leading U.S. psychology organization holds a library of more than three million publications. Until now, finding the tests and measures used in each one was a specialist's job, done by hand, because the organization's own attempts at AI had produced answers it couldn't trust. We built a pipeline of separate model steps. One step finds each test, another extracts its details, and a third keeps the citation next to it, which lets a specialist check the uncertain ones against the paper.

Before

Finding these tests by hand needed an expert.

An AI tool had already failed the team.

After
2–3 minutesper document, where it took about 3 hours
~$0.10a document, where specialist time cost $100 an hour

As the organization reported.

Finding these tests by hand needed an expert

Researchers rely on the organization to tell them which psychological tests and measures appear in which studies. Tests appear inside articles of every length and structure, and finding them meant subject-matter experts reading the archive by hand. At about $100 an hour of expert time and about three hours a document, the archive could never be covered.

An AI tool had already failed the team

Earlier in-house experiments, including a proof of concept, failed with high rates of made-up answers, so the organization was skeptical that AI could do this work at all. Its systems were also built mostly outside AWS, which made AWS-native AI tools new territory for its team.

Each answer appears with the citation that backs it

Because of that skepticism, the system attaches a citation to every test it returns and sends any uncertain extraction to a specialist for review first, so a researcher can check any answer against the paper in a few seconds.

How we built it

2 decisions, in the order we made them. Each one says what we did and why, with what it did drawn beside it.

  1. We split the work into steps that can each be checked
  2. Specialists review only the cases the system is unsure of

We split the work into steps that can each be checked

Built withOpenSearchLambdaSageMakerLangChainBedrock
Ingest and segmentExtract metadataSummarise the paperFind the testsCheck the citation
The engineering

The earlier attempts relied on one prompt asking what tests were in the paper. That single step was where they went wrong, which is why we split the work into stages.

  • Ingestion and segmentation prepare articles of very different sizes before processing starts.
  • Metadata extraction and structuring set up the context that the model calls use next.
  • The model chain breaks the work into separate steps. A summary step reads the paper first, then a step finds the psychological tests, and a last step keeps each test's citation attached, which means every extracted result can be checked against the publication.
  • OpenSearch retrieves the relevant material, a retrieval-augmented design tuned to the organization's content, while Lambda and SageMaker run the data processing.
  • LangChain coordinates the model chains, with Bedrock providing access to the model families evaluated, including Claude and Cohere, and central model testing and orchestration.

Detection and citation are kept separate, because a wrong test would then show up by its missing citation, which a researcher would notice right away.

Specialists review only the cases the system is unsure of

How sure the system is Goes through Specialist below the line, a person checks it
The engineering

Confidence scoring sends the uncertain extractions to specialists, with the publication right beside them. Their feedback feeds back into prompt optimization and changes to the extraction workflow, so the system learns from the cases it found hardest. Single sign-on (SSO) and account-level permissions control who can use the system.

What we'd tell another CTO

1The team's first AI attempt failed because people didn't trust it, which is why they built checking into every step. Each answer carries its citation, and detection is kept separate from citation, which means each one can be tested on its own.
2Confidence scoring flags the uncertain cases, and the experts correct those, feeding the corrections back into the prompts, because those cases improve the prompts the most. The team reports the volume it actually processed separately from the library's full size, because the two numbers differ.

What else changed

As the organization reported.

  • A workflow that scales across the whole repository, which makes extraction from three million publications feasible. That figure is the size of the collection, not the volume this engagement processed.
  • Every extracted test linked to its citation, with the uncertain ones reviewed by a specialist.

What comes next

Processing more of the collection through the same workflow, with the specialists' feedback continuing to refine it.

How we'd do it today

This project came before our current offers. Here's how we'd approach the same work now.

A golden set of papers whose tests the specialists already identified, plus a sealed set nobody tunes to, would gate every change, and where an AI grades answers its grades would be checked against the experts' own on unseen cases. Across three million publications most extractions are routine, so AI Inference Engineering would send them to a model the organization owns on Amazon SageMaker AI, tuned with LoRA adapters on the specialists' reviewed answers, while the hard ones still go to a frontier model through Amazon Bedrock.

Overnight, AI agents tried new prompts against the golden set. Anything that gamed the score or failed the sealed tests was dropped, and only a real improvement went to the specialists for approval.

AI Inference Engineering Modernize with AI AK AI Factory

More work like this

  1. AI~90%An ad-intelligence businesslower infrastructure costTwenty-four hours of stream. About thirty minutes worth keeping.~95% less video stored~20 minutes a run, down from 30 to 40On AWS Lambda · Bedrock · S3 · SQS
  2. AI95%A U.S. clinical-data companyfewer duplicate entries, as reportedOne patient's history could run to 2,000 pages, much of it the same visit written down again.Up to 95% extraction accuracy, with review40% faster access to structured recordsOn AWS ECS · S3 · SQS · Bedrock
  3. AI10–50×A U.S. health-data platformfaster answers when a question repeats, as reported"Is this drug covered?" was asked again and again. Each time, it paid for a fresh model call.30–40% lower cost on model processing90% less backend load from file transfersOn AWS Nova Pro · ECS · CloudFront · Bedrock

All case studies →

Have a problem like this one?

A free workshop: a few hours with the architects who'd do the work, on your own systems. No cost.

Book a free workshop