Armakuni Contact us
← All case studies

The mixing screen showed the chemicals.
It never said whether they were safe together.

A U.S. agricultural cooperative mixes five or more chemicals at a time, and its safety information lived in 142 PDFs nobody could search mid-job. Nothing stopped a dangerous combination if no experienced worker noticed it. In a four-to-five-week proof of concept (POC), we turned those Safety Data Sheets into a searchable database, read the mixing screen to give a green, yellow or red verdict, and built a chat that answers only from the cooperative's own documents. AWS funding covered the $10,000 cost.

Before

The safety data sheets sat in PDFs, away from the job.

After
Green, yellow, reda verdict on the chemicals on screen
142Safety Data Sheets in one searchable database

The safety data sheets sat in PDFs, away from the job

Every chemical the cooperative uses comes with a Safety Data Sheet covering its hazards, safe handling and what it mustn't be mixed with. All 142 were stored as PDFs. A worker who wanted to check a combination had to stop, find the right documents and read them, in the middle of a job.

The mixing application showed chemical names and quantities, but nothing connected what was on screen to what the safety sheets said. If someone loaded an incompatible combination, nothing in the system would warn them. The only protection was operator knowledge, and a question nobody could answer quickly meant stopping work or guessing.

The goal was a safety net on every screen

The long-term ambition was continuous monitoring: a desktop application on every Windows machine, catching a dangerous combination the moment it appears on screen. Before commissioning that, the cooperative needed proof that AI could read the application screen, identify the chemicals shown, and retrieve the right safety information from its own documents.

An AI agent that reads the screen and checks it against the sheets

The proof of concept tested exactly those steps, on the cooperative's own Safety Data Sheets and real application screenshots. One prepared data foundation would serve both a screenshot check and a chat for workers' questions.

How we built it

3 decisions, in the order we made them. Each one says what we did and why, with what it did drawn beside it.

  1. 142 PDFs become one database
  2. A verdict a worker can act on in a second
  3. Built on the production foundation, and tested for attack

142 PDFs become one database

Built withTextractS3AuroraMySQLOpenSearch
Safety sheet 1Safety sheet 2… 142 in all One database
The engineering

Textract extracted the information from all 142 PDFs into structured records, so no PDF has to be opened by hand again. The original documents stay in S3, with Aurora MySQL holding the chemical and document information and OpenSearch supporting retrieval. That preparation serves two interactions:

  • A screenshot-analysis workflow identifies the chemicals, quantities and proportions shown by the mixing software, then looks up the relevant safety information.
  • A chat interface takes plain-language questions, such as "what happens if I mix these two?", and retrieves the matching document content for the answer.

Because the documents are extracted once, every later interface reads the same verified data, the desktop monitor included.

A verdict a worker can act on in a second

Built withNova ProBedrockLangChainFastAPIStreamlit
A mix on the mixing screen GreensafeYellownear its limitsRedwith the corrective action
The engineering

Amazon Nova Pro on Bedrock supplies the visual interpretation and the conversational model. LangChain connects the model workflow to the document and database access, with Python and FastAPI behind a Streamlit interface for the proof of concept.

The screenshot workflow presents a color-coded assessment beside the identified chemicals:

  • Green when the combination is safe.
  • Yellow when a mix is near its limits or the combination is suboptimal, with the specific hazard warning and a suggestion, such as an alternative fertilizer.
  • Red with the corrective action.

The chat answers from the Safety Data Sheet content it retrieves, so a worker gets the cooperative's own guidance instead of whatever a general model remembers.

Why: someone mixing chemicals will act on whatever the tool says, so an answer it can't trace to a document does more harm than saying nothing.

Built on the production foundation, and tested for attack

Built withECSFargateAuroraOpenSearchS3CloudFront
The proof of conceptThe production foundationCodePipeline release pathTested for attack
The engineering

We didn't build a throwaway demo. The deployment design places ECS/Fargate application containers behind a load balancer across private subnets in two Availability Zones, with Aurora replication, OpenSearch, S3, CloudFront and WAF (with Route 53 in front), and CloudWatch and CloudTrail for operational visibility:

  • CodePipeline, CodeBuild, ECR and CodeDeploy supply the build-and-release path.
  • The same data foundation can serve later interfaces without extracting the documents again.

A separate engagement ran six days of security testing: three days of external penetration testing and three of internal vulnerability assessment.

One mix, end to end

  1. Take an example: an operator sets up a mix of five chemicals, with quantities, in the cooperative's mixing software.
  2. They capture the screen and send it to the safety check.
  3. Nova Pro reads the screenshot, identifying each chemical, its quantity and its proportion.
  4. The system looks each one up in the database built from the 142 Safety Data Sheets.
  5. The verdict comes back: yellow, with the specific hazard for two of the chemicals.
  6. The operator asks the chat what to do about it, and gets the answer from those two chemicals' own sheets.

What we'd tell another CTO

1Extract the documents once and build every interface on that, so the screenshot check, the chat and the desktop monitor all read the same verified data. In a safety tool, answer only from the source documents, with retrieval before generation every time.
2Give a verdict a person can act on at a glance, green, yellow or red with its reason, and prove it on the cooperative's own screens and documents. We built the proof of concept on the production foundation, so there's nothing to throw away.

What else changed

  • A safety chat that answers from the cooperative's own documents.
  • Infrastructure the production system can build on, across two Availability Zones, with security filtering, monitoring and a deployment pipeline, plus six days of security testing.
  • A $10,000 proof of concept, AWS-funded, at no cost to the cooperative, in four to five weeks.

What comes next

The production build: continuous monitoring on Windows desktops (watching the mixing program instead of screenshots), barcode-based chemical identification, acreage-based quantity calculations and integration with the existing mixing application. Each needs its own production validation. Reading the mixing program's database directly, in place of a screenshot, is expected to cut operating cost by up to 30%, because fewer calls go to a large model. That's a target, not a result yet.

How we'd do it today

This project came before our current offers. Here's how we'd approach the same work now.

AK AI Factory's Prove would start from a golden set of real past mixes and combinations the safety team knows are hazardous, each with its right verdict, plus a sealed set nobody tunes to, and a dangerous mix marked green would fail any change outright. What counts as yellow or red would be the safety team's decision, written down and signed off, so the verdict never rests on the model's judgment alone.

The desktop monitor's AI agent would get its own identity and only the tools its job needs, reading the screen and the database with no way to change a mix. Every verdict would be recorded with the model version and the documents it used, and a new version would reach a small share of machines first and roll back on its own if accuracy crosses a line agreed in advance.

Modernize with AI AK AI Factory

More work like this

  1. AI40%A U.S. healthcare data platformfaster document processing, as reportedA misread dosage doesn't stay on the page. It follows the patient into the next system.35% lower extraction cost25% better text-extraction accuracyOn AWS ECS · Fargate · CloudWatch · Textract
  2. AIFirstA U.S. drug testing networklabeled sensor dataset for this kind of testing, and a working prototypeCameras weren't allowed in the room. So we built a way to sense cheating without seeing anyone.24,000+ tests a day the pipeline is sized for120 testing centersOn AWS S3 · Lambda · DynamoDB · API Gateway
  3. AIParallelA Canadian legal software companyreading of every bundle, so throughput is no longer capped by headcountA new law firm arrived with 400 pages. Someone had to read every one before it went live.5–6 weeks to a working prototype that reads onboarding bundles in parallel300–400 pages per bundleOn AWS Bedrock · S3 · ECS · Fargate

All case studies →

Have a problem like this one?

A free workshop: a few hours with the architects who'd do the work, on your own systems. No cost.

Book a free workshop