What should I log from an AI agent?

The full trace: system prompt version, every model call with inputs and outputs, every tool invocation and what it returned, retries, latency, token counts and cost per step, plus a session identifier that ties it back to a user. Redact personal data at the boundary. If you can't replay a bad run from your logs, you aren't logging enough.

How many evals do I need before launch?

Start with twenty to fifty cases drawn from real usage, weighted toward the failures that would embarrass you. Run them on every prompt or model change. Coverage matters far less than the habit. A small suite that actually runs beats a large one that nobody maintains.

The core is open source and self-hostable at no cost; there is a paid cloud offering and some enterprise features under a separate licence. For most teams the self-hosted version is enough, and running it yourself means your traces, which contain user data, stay in your own infrastructure.

How do I stop agent costs from running away?

Four things, in order of payoff: a hard cap on loop iterations, prompt caching for the parts of the context that repeat, a smaller model for the steps that don't need the big one, and a per-run cost alert so you learn about a regression from a dashboard rather than a bill.

When is an agent actually ready to launch?

When you can answer three questions without opening the code: what did it do for this specific user, would today's change have broken last month's fixed case, and what did this run cost. If any answer is a shrug, you have an impressive demo rather than a product.

Agent Library · Production

The boring half: tracing, evals and cost control before you launch an agent

Before an AI agent goes to production it needs three things the demo did not: tracing of every model call and tool result, an evaluation suite that catches regressions before users do, and per-run cost tracking. Langfuse is the leading open-source, self-hostable option covering all three, and Anthropic's claude-cookbooks provides runnable patterns for tool use, retrieval and prompt caching. Both are free.

Free. 2 repositories, no payment. One email address opens the list.

The demo is the easy twenty per cent. I say that to every client and roughly none of them believe me, until the first week of real traffic.

That's when an agent that behaved perfectly in testing starts doing something inexplicable for about one customer in forty, and nobody can reconstruct what happened, because nothing was recorded.

So: tracing first, second agent later. The first thing we add to any agent project is a trace of every model call, every tool result and every retry, with the inputs attached. Not because it's good practice in the abstract, but because the alternative is debugging a non-deterministic system from a screenshot a customer sent you.

Then evals. You don't need an academic benchmark. You need maybe thirty examples of things your agent must get right, run automatically on every change, with a human reading the diffs. Teams skip this because it feels slow, then ship a prompt tweak that quietly breaks a case they fixed a month ago. A prompt change is a code change. Treat it like one.

And cost, which has a way of arriving as a surprise. Agents loop. A bug that adds two extra tool calls per run doesn't look like much until you multiply it by your user base. Track spend per run, alert on the outliers, and cap the loop. A hard iteration limit is unglamorous and has saved more than one of our projects from a memorable invoice.

This week: pick your last confusing agent failure and try to reconstruct it from your logs alone. Whatever you reach for and can't find is the first thing to start recording.

Where to send the 2 repositories

Give me an email address and the full list opens on this page straight away, with a copy in your inbox you can search later, plus a note whenever we publish something new on AI and where mobile development is going. No payment, now or ever. That's the only trade.

Production: the boring half nobody posts about

The demo is the easy 20%. These two are what stand between a working prototype and something you would let a client near.

2 repositories, with the notes on each. Enter your email above to read them.

The master list: all 100 repositories

The free list above is the short version, and every repository on it is in here too. The Gods Repo Library is the whole catalogue: 100 repositories across 12 categories, each with a written note on what it is for and when to reach for it, and it carries the AI optimisation half, inference, serving, quantisation and evaluation, that the free list barely touches. Every figure in it is read from the GitHub API rather than typed by hand, and nothing archived or idle survives the build. Verified 2026-09-23.

Orchestration: where the agent lives19
Coding agents you can point at your own repo6
MCP, tools and the integration layer10
Browser and computer control5
Retrieval: giving the model your data9
Memory and vector stores7
Evaluation, observability and guardrails11
Prompt and output optimisation5
Inference, serving and routing10
Fine-tuning, quantisation and compression7
Getting the data in5
Courses, cookbooks and reference6

$9.99once, for all 100 repositories, sent to your inbox

All 100 repositories with a written note on each, emailed to you
88 of them are beyond the free list above
A permanent link to the live version, refreshed against the GitHub API
No subscription and no account, just the library

Before you buy, the terms of this sale

The full catalogue is delivered to the email address I give, immediately, the moment my payment clears.
Because it is delivered in full straight away, all sales are final: no refunds, no partial refunds, no exceptions, for any reason.
I am asking for that immediate delivery and I give up any cancellation or cooling-off period that would otherwise apply to digital content.
If the email does not reach me, Frenchy Digital will re-send it to my address for as long as I need, and that re-send is the remedy rather than a refund.
The repositories listed are third-party open-source projects. Their availability and their licences are theirs to change, and I will check any licence myself before using it commercially.

I have read and accept the terms above. I understand this purchase is final and no refund is available.

The same terms are in full in our terms and conditions, and you will be asked to accept them once more on the Stripe payment page.

You can change the address at checkout, and that is where it is delivered. Secure checkout by Stripe, and we never see your card.

Also in the library: the Agency CRM Ops Skill

A Claude Code skill that runs an agency’s client operations: a pipeline that only holds real prospects, solicitor detection, a lawful marketing list, meeting recordings filed one folder per client, and inbox cleanup that never touches your favorites. $20 once.

See what the skill does

Tip if it helped. Only if it helped.

The library costs nothing and stays that way. You already have it, whether or not you type anything here. If it saved you an afternoon, name your own amount. It goes toward the time we spend testing these so you don't have to.

Anything from $2. You can still change it on the next screen. Secure checkout by Stripe, and we never see your card.

Our partners

Tools we use or recommend to clients, with the offer each one gives new customers. These are referral links: if you sign up, we may earn a commission at no cost to you. Offers checked September 2026 and set by each company, so confirm on their page.