01 — ASSESS
Understand the problem
Interview the people doing the task today. Record what drives their decisions and what the process costs now.
SERVICES — AI SYSTEMS · METHOD
Evals first. Then everything else.
We build LLM systems the way you would build anything that handles money or decisions: measured from the first week. Every change is judged by tests that are tied to what a mistake costs your business, not by whether the output looks better.
01 — BEFORE ANY CODE
The first meeting
Four conversations that give us the problem, the examples and the cost model.
Read more →02 — THE PROJECT
Seven stages
From a rough first version to a system that keeps improving in production.
Read more →03 — IN PRACTICE
A worked example
Checking payment receipts for an East African finance team.
Read more →THE FIRST MEETINGWHERE THE EVALS BEGIN
We spend the first meeting on your workflow, not our pitch.
A useful LLM system starts with the people who do the work today, the examples they decide on, and what a wrong decision costs. The first meeting is how we get all three.
You will do most of the talking. We will ask questions and play back what we heard at the end.
01 — PEOPLE AND CONTEXT
Who is involved
- Who is in the room, and who makes the final decision?
- Who does this work today? They become our domain experts.
- Which other teams are affected: finance, compliance, IT?
02 — THE WORKFLOW TODAY
How the work is done
- Walk us through it step by step. What do people look at to decide?
- How many items a month, and what does each one cost to handle?
- What happens when it goes wrong, and what does that cost?
- What examples exist: real inputs, past decisions, labelled cases?
- What constrains it: payment rails, regulator, data residency, connectivity?
03 — WHAT SUCCESS LOOKS LIKE
What is worth paying for
- What result would make this worth doing, in money or time saved?
- Which mistake is worse: a false alarm or a miss? What error rate is acceptable?
- What is the budget per item, and how fast does an answer need to be?
- Have you tried AI or automation on this before? What happened?
- What are the timeline and budget, and what do you need to see in a proposal?
04 — PLAYBACK AND NEXT STEPS
What we agree before we leave
- We summarise what we heard, and you correct what we got wrong.
- We agree what the written recommendation will cover.
- We agree the three things below, with dates.
WHAT WE ASK YOU FOR
A
20–50 real examples
Actual inputs from the workflow, with the decisions your team made on them if you have them.
B
An hour or two of an expert
Someone who does the work today, to review our first outputs and explain their calls.
C
A date for our recommendation
A written assessment of whether an LLM helps here, what a first version looks like, and what it should save.
THE SEVEN STAGESASSESS · PROTOTYPE · EVALUATE · SHIP
A rough version early, measured honestly, improved where it pays.
Most projects start without clean labelled data or a complete picture of the problem. The process is built for that. Understanding the problem never stops: what we learn at every stage feeds back into the first.
02 — ASSESS
Gather examples
Collect real inputs, labelled or not. The test set grows over the project, with your experts.
03 — PROTOTYPE
Build a V0, end to end
Right inputs in, right shape out. Usually one prompt, one model and structured output. Quality comes later.
04 — EVALUATE
Label data, write first evals
Run V0 on real inputs and have experts correct its output. Their corrections become the first ground truth.
05 — EVALUATE
Tie evals to money
A simple cost model turns each score into a figure, so we fix what costs you most. Not every metric matters.
06 — SHIP
Improve, then raise the bar
Work back from wrong decisions, re-run the same evals, compare. When the system passes, write harder evals.
07 — SHIP
Keep QA running live
Log, sample real traffic for expert review, capture corrections. Ship a new version only when it beats the old one.
WORKED EXAMPLEPAYMENT RECEIPT CHECKS · ILLUSTRATIVE
Checking payment receipts for a finance team in Addis Ababa.
Customers pay invoices by mobile money and bank transfer, then send a screenshot or photo of the receipt. A finance team checks each one against the invoice: payer, amount, reference and date. Mismatches go to a senior reviewer.
Receipts arrive in Amharic and English, as screenshots and phone photos, sometimes with handwritten notes. Figures below are illustrative, not from a client.
THE V0: TWO STEPS
STEP 1
Extract
Read the receipt image into structured fields: payer, amount, reference, date, rail.
STEP 2
Decide
Compare the fields with the invoice and decide: approve, or send to a reviewer, with the reason.
THREE THINGS WE MEASURE
EXTRACTION
Did we read it right?
Given the image, how many fields match what an expert reads?
JUDGEMENT
Was the call right, given what it saw?
Given the extracted fields, even wrong ones, did step 2 make the correct decision? We don’t blame it for step 1’s errors.
OUTCOME
Was the final answer right?
Given the image, did the receipt end up approved or escalated correctly? This is the number the business feels.
WHAT THE COST MODEL TELLS US
ASSUMPTIONS
- 240,000 receipts a year; 4% need a reviewer
- A reviewer check costs $1.50
- A bad receipt that slips through costs $120 on average
MISSED BAD RECEIPTS$11,520a year for each percentage point of bad receipts we approve. The top priority.
FALSE ALARMS$3,456a year for each percentage point of good receipts we escalate. Worth fixing second.
PAYER NAME SPELLING$0when Amharic names are transliterated differently but the decision is unchanged. The eval looks bad; it doesn’t matter.
WHEN THE EVALS SAY IT ISN’T GOOD ENOUGHTRY IN THIS ORDER
The cheapest fix first. Fine-tuning last.
Each step down costs more to build and maintain. Fine-tuning on a weak prompt locks the weakness in, so we only move down when the evals show we need to.
01
Model
A stronger model, or more reasoning budget.
02
Prompt
Clearer instructions and explicit rules.
03
Examples and context
Worked examples, or retrieval that pulls in similar cases.
04
Tools
Lookups in your systems, APIs and databases.
05
Helper models
Sub-tasks, guardrails, or several models whose answers are combined.
06
Fine-tuning
On labelled data or eval graders, never on the eval set itself.
Once quality is where it needs to be, the same evals let us move to smaller, cheaper, faster models without guessing what we lose.
WHAT WE HOLD TOSIX PRINCIPLES
01
Evals are the backbone
Not a final check. They are written in week one and run on every change.
02
End to end beats polished
A rough system that runs from input to decision teaches more than one perfect part.
03
Experts’ time is scarce
We ask for small, regular reviews and use every correction twice: as a test and as guidance.
04
Every metric has a price
If a score doesn’t connect to money or time, we don’t optimise it yet.
05
Judge each step fairly
Each step is scored on the inputs it actually received, and the whole system on the outcome.
06
Passing means write harder tests
When the system passes, we add more data and more reviewers, and raise the bar.
Credit: this method is adapted from Eval-Driven System Design: From Prototype to Production, OpenAI Cookbook, a collaboration between OpenAI and Fractional. © 2025 OpenAI, MIT License. The worked example is our own.
Read the original →