The boring half: tracing, evals and cost control before you launch an agent
Before an AI agent goes to production it needs three things the demo did not: tracing of every model call and tool result, an evaluation suite that catches regressions before users do, and per-run cost tracking. Langfuse is the leading open-source, self-hostable option covering all three, and Anthropic's claude-cookbooks provides runnable patterns for tool use, retrieval and prompt caching. Both are free.
Free. 2 repositories, no paywall. Tip only if it helps.
The demo is the easy twenty per cent. I say that to every client and roughly none of them believe me, until the first week of real traffic.
That's when an agent that behaved perfectly in testing starts doing something inexplicable for about one customer in forty, and nobody can reconstruct what happened, because nothing was recorded.
So: tracing first, second agent later. The first thing we add to any agent project is a trace of every model call, every tool result and every retry, with the inputs attached. Not because it's good practice in the abstract, but because the alternative is debugging a non-deterministic system from a screenshot a customer sent you.
Then evals. You don't need an academic benchmark. You need maybe thirty examples of things your agent must get right, run automatically on every change, with a human reading the diffs. Teams skip this because it feels slow, then ship a prompt tweak that quietly breaks a case they fixed a month ago. A prompt change is a code change. Treat it like one.
And cost, which has a way of arriving as a surprise. Agents loop. A bug that adds two extra tool calls per run doesn't look like much until you multiply it by your user base. Track spend per run, alert on the outliers, and cap the loop. A hard iteration limit is unglamorous and has saved more than one of our projects from a memorable invoice.
This week: pick your last confusing agent failure and try to reconstruct it from your logs alone. Whatever you reach for and can't find is the first thing to start recording.
Production: the boring half nobody posts about
The demo is the easy 20%. These two are what stand between a working prototype and something you would let a client near.
- langfuse/langfuse34,605· TypeScript · MIT / EE
Traces, evals and cost tracking you can self-host. The first thing we add to a project, before the second agent.
- anthropics/claude-cookbooks52,690· Jupyter Notebook · MIT
Runnable notebooks straight from Anthropic: tool use, retrieval, caching. Less theory, more copy-paste that works.
Star counts are a snapshot taken 2026-09-14, not live figures. Licences are as published by each project, so check them yourself before you ship commercially.
Want the whole list in your inbox?
Every repository above, with the notes, in one email you can search later, plus a note from me whenever we publish something new on AI and where mobile development is going. That's the only trade.
Tip if it helped. Only if it helped.
The library is free and stays free. You already have it, whether or not you click anything here. If it saved you an afternoon, you can name your own amount. It goes toward the time we spend testing these so you don't have to.
You can change the amount on the next screen. Secure checkout by Stripe, and we never see your card.
Questions people actually ask
Ready to Build Your Application?
Let's discuss your project. Our team will help you build solutions that transform your business.