Field Notes
The rest of this site is method. This page is experience - what actually happened building two production AI systems, including the part that cost weeks and the assumption that turned out to be wrong.
Last reviewed: August 2026
The Projects
The first is an agentic supply chain intelligence platform: it ingests purchasing data and contracts, surfaces variance against contracted pricing, scores compliance against committed spend, and flags risk before it becomes a write-off. Python and FastAPI over PostgreSQL, with self-hosted inference and a two-pass contract extraction step that attaches confidence scores to what it pulls out.
The second is a cost-optimized document AI service for a regulated income-verification platform - extracting structured data from paystubs, tax forms, and eligibility documents. It exists because the platform was paying full frontier-model prices for a level of capability most documents never needed.
The technical write-ups live on rjl.dev. What follows is not that. These are the things I would tell someone before they started, which is a different list from the things you put in an architecture document.
What I Got Wrong: Language Drift
The expensive mistake was not a bug. It was letting the implementation language drift.
The specification did not pin the stack tightly enough, so Java took hold alongside Python. Nothing dramatic announced itself - each individual step looked reasonable in isolation, and an agent will happily produce competent code in whichever language the immediate context suggests. By the time it was obvious, there were two languages in a codebase that only ever needed one. Consolidating the Java work back into Python cost a few weeks and bought nothing.
The fix is embarrassingly small: state the language, and state that it is the only one. That single constraint in the spec would have prevented all of it. This is what context engineering means in practice - not elaborate prompting, but noticing which of your assumptions you never actually wrote down. I assumed "obviously this is a Python project." The agent had no way to know that was a rule rather than a coincidence.
It is also a good argument for reviewing at the level of direction and not just diffs. Every individual change was fine. The trend was the problem, and trends do not show up in a single Develop phase review.
What Surprised Me: The Frontier Model Was Not the Answer
On both systems the model doing most of the work was not a frontier model. It was an open-weights Qwen model, self-hosted, with lighter alternates available for cheaper inference. It worked great.
That was not the plan going in. The reflex is to reach for the strongest model available and treat everything else as a downgrade. But most of what these systems ask a model to do is structured extraction and scoring against known criteria - bounded, repetitive, well-specified work - and that is not what frontier models are uniquely good at. Self-hosting also meant sensitive data stayed inside infrastructure I controlled rather than being shipped to an external endpoint, which on both problems was worth more than a few points of general capability.
The second project is the refined version of the lesson, and it is more interesting than "use cheaper models." It routes by difficulty: a confidence-gated cascade where self-hosted models handle the common path and a frontier model is the escalation for genuinely ambiguous documents. The question stops being which model and becomes which model for which document - and the frontier model still earns its keep, just on a much smaller slice of the traffic.
Two guardrails made that safe to do. The first is a hard rule: cost optimization that quietly degrades quality is not a win, it is a liability moved off the invoice and onto the applicant. Cheaper is only cheaper if accuracy holds, which means the cascade is gated on evaluation rather than on hope. The second is that the cheapest inference is the one you never make - blur detection, deskew, rotation and page-count checks reject unusable documents before any model sees them.
The lesson I would generalize: "best on a benchmark" and "right for this workload" are different questions, and only the second one matters on a given project. It is exactly why the selection guide on this site is a framework rather than a ranking. A ranking would have told me to use something else.
What I Changed: Plan First, Every Time
The habit that stuck: always plan before executing. Never hand an agent a task and let it start editing.
Have it produce a plan, read the plan, correct the plan, and only then let it work. A wrong plan costs one paragraph to fix. A wrong implementation costs a review cycle, a revert, and the quiet risk that something incorrect survives because it looked plausible in a large diff. The cheapest place to catch a misunderstanding is before any code exists.
This is just the Ideate phase applied at the scale of a single task, which is the part I did not appreciate until I had skipped it enough times. The five phases are not only a project-shaped structure. The same loop runs inside one afternoon's work.
What It Says About the Framework
All three lessons land in the same place, which is the honest reason this site exists. The language drift was a specification failure that surfaced during construction. The model choices were tooling decisions that a leaderboard would have gotten wrong. Planning first is the discipline that makes the other two catchable at all.
None of them were about the AI being insufficiently capable. The capability was there. What was missing, in each case, was a constraint I had not written down, a question I had not asked, or a step I had skipped because skipping it felt faster. That is the whole argument for having a method rather than a set of habits.
Frequently Asked Questions
Not always. On both of these systems the model doing most of the work was a self-hosted open-weights Qwen model, and it did the job well. The better pattern is not "avoid frontier models" but routing by difficulty: a confidence-gated cascade where cheaper models handle the common path and a frontier model is the escalation for genuinely ambiguous cases. Just gate it on evaluation, because cost optimization that quietly degrades quality is a liability moved off the invoice rather than a win.
Letting the implementation language drift. The specification did not pin the stack tightly enough, Java took hold alongside Python, and consolidating back down to one language cost a few weeks that bought nothing. It was never hard to fix, just tedious - and entirely preventable by writing one more constraint into the spec.
Planning before executing, every time. Having the agent produce a plan and reviewing that plan is far cheaper than reviewing a finished diff that went the wrong direction. It is the Ideate phase applied at the scale of a single task.
Method, Then Practice
These notes are one project's worth of evidence. The framework is what generalizes.