Skip to main contentSkip to footer

    Top Rated & Verified

    Top Clutch App Development Company Black Owned United StatesTop Clutch Java Developers France 2026Top Clutch Service Line Blind Company Black Owned 2026Top Clutch App Development Company Minority Owned 2026Top Clutch Web Developers Black Owned 2026Top Clutch App Development Company Black Owned 2026Top Clutch Flutter Developers France 2026Top Clutch Health & Wellness App Developers France 2026Top Clutch Swift Company France 2026Top Clutch Machine Learning Company France 2026Top Clutch Chatbot Company France 2026Top Clutch Artificial Intelligence Company France 2026Top Clutch App Development Company Minority Owned Los Angeles
    Back to Blog
    Modernization
    August 9, 2026
    26 min read

    Legacy System Modernizationwith AI Agents in 2026

    Written for the CTO sitting on a system nobody wants to touch. Where AI agents genuinely help on legacy work, where the claims outrun the evidence, what the METR study actually measured, and how to sequence a modernization program that does not depend on any of it being true.

    Legacy system modernization with AI agents in 2026 — code archaeology, characterization tests, and incremental strangler-fig migration
    19% slower
    Measured effect of AI on experienced devs in their own repos
    METR RCT, July 2025 — now labeled historical by METR
    >70%
    Mainframe exit projects forecast to fail on GenAI overestimation
    Gartner prediction, June 18 2026 — not a measurement
    ~64%
    Frontier model resolve rate on fresh, decontaminated coding tasks
    SWE-rebench, tasks collected May–Jul 2026
    $80k–$200k
    Multi-workflow agent platform with integrations
    Frenchy Digital scoping, 2026

    Key Takeaways

    • No credible, independently measured productivity number exists for AI-assisted COBOL or mainframe modernization. Treat any specific 'X% faster migration' claim as unsourced until someone shows you the methodology.
    • METR's randomized trial — 16 experienced developers, 246 real issues in their own million-line repositories — measured a 19% slowdown against a forecast 24% speedup and a post-hoc belief of 20% faster. METR now labels the study historical and changed its design in February 2026; cite it as a 2025 finding about 2025 tools.
    • The lesson is not that AI makes developers slower. It is that perceived speedup is an unreliable guide on unfamiliar, high-context codebases — which is exactly what legacy modernization is.
    • Where agents genuinely help is comprehension: code archaeology, dependency mapping, documentation of undocumented systems, characterization test generation, business-rule extraction into reviewable specs, and incremental strangler-fig migration.
    • The integration constraint decides most projects. Legacy systems rarely have APIs, so the agent's write path is the hard problem. Computer-use is reliable on structured single-site flows and unreliable on long-horizon work — both flagship consumer browser agents were shut down, in 2025 and 2026.
    • Human-in-the-loop task queues are often the correct architecture, not a failure. They convert an unreliable write path into a reviewed one.
    • Vendor coding benchmarks say almost nothing about your undocumented 1980s codebase: models score around 64% on fresh decontaminated tasks against the ~95% claimed on contaminated public sets. Build a private eval from real tickets.
    • Sequence it: comprehension first, tests second, incremental migration third. Never a big-bang rewrite.
    • Frenchy Digital cost bands: discovery and architecture review $9k–$22k; single production agent $30k–$80k; multi-workflow platform $80k–$200k; enterprise or regulated build $200k–$450k+.

    The Evidence Gap, Stated Plainly

    Start with the uncomfortable part, because everything else in this article depends on it. As of August 2026 there is no credible, independently measured productivity number for AI-assisted COBOL or mainframe modernization. Not a lower one than vendors claim. None. No randomized trial, no controlled field study, no auditable multi-customer dataset. The category has been sold aggressively for two years and measured by nobody outside the companies selling it.

    That is a strange thing to write in an article whose commercial purpose is to sell modernization work. It is also the reason this article is worth reading. If you are a CTO with a system that runs the business and that nobody wants to open, the most valuable thing anyone can tell you is which claims have evidence behind them and which do not — because you are going to be shown a lot of the second kind.

    The most telling admission comes from inside the ecosystem that benefits most. IBM's mainframe modernization tooling is the most serious product in this space, and IBM's own partner ecosystem concedes the point directly: tools alone do not deliver enterprise modernization outcomes. That is a systems integrator saying, in public, that the software you are being sold is not the thing that determines whether the program works.

    The rule to apply to every deck you are shown:any specific “X% faster migration” claim is unsourced until someone shows you the methodology, the baseline, the sample size, and who did the measuring. A customer logo is a testimonial. A case study written by the vendor is a marketing artifact. Neither is a measurement.

    There is one widely cited 2026 figure pointing the other direction, and it deserves precise labelling too. In June 2026 Gartner published a prediction that more than 70% of mainframe exit projects will fail due to overestimation of generative AI's capabilities. That is a Gartner Predicts statement — an analyst forecast, not a survey result and not a measurement of completed projects. It should not be repeated as data. It is, however, directionally consistent with the complete absence of evidence on the optimistic side, which is why it is worth knowing about.

    The better-grounded numbers describe enterprise AI generally rather than modernization specifically, and they are sobering enough. Gartner's April 2026 survey of 782 infrastructure and operations leaders, fielded November to December 2025, found that only 28% of AI use cases fully succeed and meet ROI expectations, while 20% fail outright— leaving roughly half in partial-success territory. Note that this is materially less pessimistic than the “80% of AI projects fail” folklore, and it is real measurement rather than prediction. Skills gaps and data quality were each cited by 38% of respondents as the blocker.

    Stanford HAI's 2026 AI Index adds the scaling picture: 88% of organizations use AI in at least one function, fewer than 10% have fully scaled it in any single function, and only 29% report significant ROI. Deloitte's 2026 State of AI in the Enterprise (n=3,235 senior leaders across 24 countries, fielded August to September 2025) supplies the cleanest expectation-versus-reality figure available: 74% hope to grow revenue through AI; 20% already are.

    We will tell you what the evidence supports before we tell you what we would build. On legacy modernization the honest answer is that the tooling is real, the comprehension gains are real, and the migration-speed claims are not yet measurable. Plan the program so that it works either way.

    Frenchy Digital engagement principle

    The METR Study, Reported Precisely — and Its Essential Caveat

    This is the most important piece of evidence in the article, and it is misquoted constantly in both directions, so here it is exactly.

    In July 2025, METR published a randomized controlled trial measuring the effect of AI tools on experienced open-source developers. The design matters as much as the result. 16 experienced developers worked on 246 real issues in their own repositories — mature projects averaging more than 22,000 GitHub stars and over a million lines of code, with the contribution standards that implies. Issues were randomized to allow or forbid AI assistance. The tooling was Cursor Pro with the Claude Sonnet models of that period.

    The result: developers were 19% slower on issues where AI was allowed. Before starting, they forecast a 24% speedup. After finishing — having done the work and seen the clock — they still estimated AI had made them 20% faster. That is roughly a 39-point perception gap against a measured slowdown. The perception gap is the finding, more than the slowdown itself.

    Now the caveat, which is not optional. METR itself now labels this study historical, notes explicitly that it does not necessarily reflect current tools or workflows, and changed its experiment design in February 2026. Anyone citing 19% as a fact about AI coding tools in 2026 is misusing it — including anyone citing it in support of the argument this article is making. It is a 2025 finding about 2025 tools. Cite it that way or do not cite it.

    So what survives the caveat? Not “AI makes developers slower.” The tools have changed, the harnesses have changed, and the study was never designed to hold across model generations. What survives is narrower and considerably more useful to you:

    • Perceived speedup is not evidence of speedup: The developers in the trial were not fooled by a marketing claim. They were fooled by their own experience of doing the work. Self-report was wrong by nearly 40 points in the direction people wanted it to be wrong.
    • The effect was largest exactly where legacy work lives: High-context, unfamiliar-to-the-model codebases with strict quality bars and a maintainer who knows where the bodies are buried. That is the profile of every legacy modernization engagement.
    • Model capability was not the binding constraint: The developers spent time reviewing, correcting, and re-prompting. On a large old codebase the cost of verifying an AI-proposed change is the dominant cost, and it scales with how little anyone understands the system.
    • The remedy is measurement, not abstinence: The correct response to an unreliable perception signal is an instrument, not a policy. Baseline your change lead time and defect rate before the program starts, because afterwards nobody will be able to tell you honestly whether it helped.

    Two adjacent data points give the same signal from a different angle, and both are large-sample. Google's DORA research on AI-assisted development (roughly 5,000 professionals, fielded mid-2025) found 90% using AI at work while 30% reported little-to-no trust in AI-generated code. Stack Overflow's 2025 developer survey (n≈49,000) found 46% distrust AI accuracy against 33% who trust it, only 3.1% highly trust it, 66% cite output that is “almost right, but not quite” as the top frustration, and 45% say debugging AI-generated code takes longer than writing it themselves. Note there is no 2026 edition of that survey; those figures are 2025.

    “Almost right, but not quite” is the exact failure mode that makes legacy work dangerous. In a greenfield system, an almost-right change fails a test you wrote last week. In a forty-year-old system with no tests, an almost-right change ships.

    Claim you will be shownWhat the evidence actually supportsWhat to ask for instead
    AI cuts legacy migration time by X%No independently measured productivity number exists for AI-assisted COBOL or mainframe modernization, from any source, as of August 2026.Ask for the methodology, the baseline, the N, and who measured it. If the answer is a customer logo, it is a testimonial rather than a measurement.
    Our tool automatically converts COBOL to modern JavaVendor-stated capability. IBM watsonx Code Assistant for Z performs COBOL-to-Java and PL/I-to-Java transformation with application analysis; no independent productivity evaluation has been published.Evaluate on your own code, with your own characterization tests, on a module you already understand. Compare against a human baseline you actually timed.
    AI coding assistants make developers dramatically fasterThe only randomized controlled trial on experienced developers in their own large repositories measured a 19% slowdown (METR, July 2025). METR labels that study historical and changed its design in February 2026.Do not use the 19% as a current claim, and do not use vendor speedup claims as one either. Measure on your team, on your code.
    Mainframe exit is now feasible because of generative AIGartner published a June 2026 prediction that more than 70% of mainframe exit projects will fail due to overestimation of generative AI's capabilities. That is an analyst forecast, not a measurement.Treat exit as a multi-year program with incremental value gates, not a project with an end date driven by tooling.
    Model X scores 95% on coding benchmarks so it will handle our codeOn fresh, decontaminated tasks collected May–July 2026, frontier models land around 64%. The official SWE-bench Verified board has been academic-only since November 2025 and is frozen at 79.2%.Build a private eval from your own closed tickets. Public benchmark position is not transferable evidence.
    Tools will deliver the modernizationIBM's own partner ecosystem concedes that tools alone do not deliver enterprise modernization outcomes.Budget for domain experts, test infrastructure, and organizational change alongside the tooling line item. The tooling is the cheapest part.

    Common legacy-modernization claims mapped against the available evidence — Frenchy Digital, August 2026.

    Where AI Agents Genuinely Help: Comprehension, Not Rewriting

    Reframe the problem and the value becomes obvious. The reason nobody wants to touch your legacy system is not that the language is old. It is that the system's behavior exists only in the code, and nobody has read all of the code. The original authors are gone. The documentation describes a version from three acquisitions ago. The business rules are distributed across copybooks, batch jobs, stored procedures, and one screen that only Denise knows how to reach.

    That is a comprehension problem, and comprehension is what these models are genuinely good at. Reading a large corpus and producing structured summaries with citations back to the source is close to the core competence. Producing correct novel code against undocumented constraints is not.

    Every use below shares three properties that make it safe: the task is read-heavy, the output is a reviewable artifact rather than a production change, and being wrong is visible rather than silent. That is the filter to apply to any proposed agent use on a legacy system.

    UseWhat the agent doesWhy it worksWhat still needs a human
    Code archaeology and dependency mappingAgent reads across the codebase and produces a call graph, data-flow map, and inventory of entry points, batch jobs, and copybooks or shared schemas.The task is read-only, the output is reviewable, and errors are visible rather than silent. Retrieval and summarization are what these models are actually good at.A human confirms the map against operational reality — which jobs still run, which endpoints still receive traffic, which tables are actually written.
    Documentation for undocumented systemsModule-by-module first-draft documentation: what it appears to do, what it reads and writes, what its error paths are, and what is unclear.Nobody has to be right on the first pass. A wrong draft that a maintainer corrects in ten minutes still beats a blank page that has been blank for fifteen years.A domain expert corrects it. Unreviewed generated documentation is worse than none, because it will be trusted.
    Characterization test generationTests that pin current behavior — including behavior that is arguably wrong — so that any change surfaces as a diff rather than an incident.This is the single highest-leverage use. It converts an untouchable system into a system you can change, and the safety net is verifiable by running it.Humans decide which behaviors are contracts and which are bugs. The agent cannot know which 1987 rounding rule is load-bearing.
    Business-rule extraction into specificationsPulling embedded rules — eligibility conditions, pricing tiers, batch cutoffs, tax logic — out of code and into a reviewable specification document.It converts an implicit asset trapped in a dying language into an explicit one the business can argue about. The specification outlives the migration.A domain expert validates each rule. This is where the real value of a modernization program usually turns out to sit.
    Incremental strangler-fig migrationAgent-assisted implementation of one seam at a time: a facade in front of the legacy system, one capability reimplemented behind it, traffic shifted, old path retired.Every step is small, tested, and independently revertible. Blast radius stays bounded, and value ships before the program completes.Humans choose the seams and own the cutover decision. Seam selection is architecture, not code generation.
    Ticket triage and change-impact analysisGiven a ticket, the agent locates the relevant code, lists probable blast radius, and drafts a change plan with the tests that should be added.It compresses the search phase, which is where most time goes in an unfamiliar system, without giving the agent write access.Engineers reject or adjust the plan. Impact analysis is a hypothesis, not a guarantee.

    Legacy modernization tasks where AI agents produce reliable value, and the human role in each — Frenchy Digital, 2026.

    If you do one thing: generate characterization tests. Not unit tests that assert what the code should do — characterization tests that assert what it currently does, including behavior that is arguably wrong. That is the artifact that converts an untouchable system into a changeable one, and it is the only artifact in this article whose correctness you can verify mechanically by running it against the system you already have.

    Business-rule extraction deserves its own note because it is where the durable value usually sits. A rule like “accounts flagged before the 1998 migration are exempt from the late fee unless the branch code is in this list” is a business asset currently trapped inside a dying language. Getting it into a specification a domain expert can read, argue with, and correct is valuable whether or not you ever migrate — and it is the input that makes the migration possible at all. Several of our engagements have produced more organizational value from the extracted specification than from the code that was eventually written.

    One technical constraint governs all of this work: context. Anthropic's engineering guidance on context describes context rot — accuracy degrading as context grows, well before the window fills — and frames context as an attention budget rather than a storage budget. On a million-line codebase this is not a theoretical concern. The approaches that work are just-in-time retrieval instead of preloading, compaction that preserves architectural decisions and unresolved questions, structured note-taking to external files that survive context resets, and sub-agent isolation where a searching sub-agent holds detailed context privately and returns a condensed summary of roughly a thousand to two thousand tokens.

    A note on how many agents you need

    The multi-agent question comes up immediately on codebase work, and the honest answer from both camps points the same way for this use case. Anthropic's own multi-agent research write-upstates plainly that coding has fewer parallelizable components than research, and that multi-agent architectures are unsuitable where agents need shared context or heavy interdependency. Cognition's widely read counter-argument makes the same point from the other side: actions carry implicit decisions, and conflicting decisions from parallel agents produce incoherent results.

    Legacy modernization is deep-and-narrow shared-state work, not wide-and-shallow independent-thread work. Fan-out is genuinely useful for the read-only archaeology phase — many independent searches over disjoint modules. It is a liability once anything is being changed. And the economics matter: Anthropic reports agents using roughly 4× the tokens of a chat interaction and multi-agent systems roughly 15×.

    The Automated Translation Pitch, and Why It Underperforms the Demo

    Wholesale automated language translation — point the tool at COBOL, receive Java — is the most seductive pitch in this category and the one with the thinnest evidence. It is seductive because it appears to solve the entire problem in one purchase, and because a demo on a well-structured module genuinely looks impressive.

    The failure mode is not syntax. Mechanical translation of syntax is a solved problem and has been for years. The failure modes are subtler and all of them are expensive:

    • Translated structure preserves the wrong things: You get Java that faithfully reproduces the shape of decisions made under 1985 constraints — fixed-width records, batch windows, storage economics that no longer exist. It compiles. It is not modern software; it is COBOL with braces, and now nobody wants to touch it either.
    • Implicit behavior does not survive: Numeric precision, rounding, collation order, date handling, null semantics, and overflow behavior differ between runtimes. A translation that is 99.9% behaviorally identical is a translation that is wrong on the 0.1% of transactions that will end up in a regulator's letter.
    • Dead code gets translated too: Without a live-versus-dead analysis grounded in production telemetry, you pay to translate, test, and maintain code that has not executed since 2011 — and you inherit its bugs into the new system.
    • The test gap is inherited, not fixed: If the source had no tests, the translation has no tests. You now have two untested systems and a cutover decision, which is strictly worse than the one untested system you started with.
    • Verification cost dominates: Somebody has to confirm the translation is correct, and that somebody needs to understand the original. The scarce resource was never the typing.

    None of this means the tooling is useless. It means the tooling is a component in a program whose critical path is comprehension and verification, not code production. Used after characterization tests exist, module by module, with a domain expert reviewing extracted rules, automated translation is a real accelerator on the part of the work that was never the bottleneck. Used as the program itself, it produces a second system nobody understands.

    A migration you cannot verify is not a migration, it is a bet. Build the characterization suite first and the translation tooling becomes genuinely useful. Skip it and no amount of model capability tells you whether the output is correct.

    Frenchy Digital architecture principle

    The Tooling Landscape as of August 2026

    Here is the honest state of the tools, with the evidentiary status of each attached. Everything marked vendor-stated is a capability claim from the company selling it, not an independent evaluation — that distinction is the whole point of the table.

    Tool or categoryWhat it doesEvidentiary statusHow to use it
    IBM watsonx Code Assistant for ZCOBOL-to-Java and PL/I-to-Java transformation with application analysis; release 2.8 added an agentic workflow in 2026[VENDOR] Capability claims are vendor-stated. No independent productivity measurement published.The most serious tooling in the category. Evaluate on your own modules with your own tests before committing a program to it.
    IBM Project BobPreviewed at TechXchange 2025 for general availability in 2026[VENDOR] Preview-stage. Nothing independently evaluated exists.Track it; do not plan a roadmap around a preview.
    General-purpose coding agentsRead, plan, edit, run tests, iterate across a repositoryPublic benchmark scores are largely contaminated (see the evaluation section). Real-world performance on undocumented legacy code is unmeasured.Genuinely useful for comprehension and test generation. Constrain write access; require review on every diff.
    Static analysis and dependency toolingDeterministic call graphs, dead-code detection, data-flow analysisDeterministic and verifiable. Not AI, and that is the point.Run this first. It is cheaper, faster, and reproducible, and it gives the agent a factual scaffold to work from.
    Screen and terminal automationDriving a legacy UI or 3270-style terminal session programmaticallyMature category with decades of production use. Brittle against interface change.Reliable when the interface is frozen. Every change to the legacy UI is a potential outage in your automation.
    Computer-use agentsA model driving a GUI by looking at screenshots and issuing clicks and keystrokesReliable on structured, single-site, short-horizon flows with a stable interface. Unreliable on long-horizon work. Both flagship consumer browser agents were shut down — Operator on Aug 31 2025, Project Mariner on May 4 2026.A last resort for the write path, and only behind approval gates. Never for unrecoverable actions.
    Durable workflow enginesCrash-resume and replay for long-running processes; Temporal's official OpenAI Agents SDK integration reached GA on March 23, 2026Deterministic infrastructure, not agent orchestration semantics.Underrated for migration work. A multi-day migration run that cannot resume from a crash is a migration run you will do twice.

    Legacy modernization tooling and the evidentiary standing of each capability claim — as of August 2026.

    On IBM specifically. watsonx Code Assistant for Z performs COBOL-to-Java and PL/I-to-Java transformation alongside application analysis, and release 2.8 added an agentic workflow in 2026. IBM also previewed Project Bob at TechXchange 2025 for general availability in 2026. These are the vendor's own descriptions of what the products do. IBM has published no independent productivity measurement, and neither has anyone else. That does not make the tooling bad — it is the most serious offering in the category and it is worth evaluating seriously. It makes the tooling unproven, which is a different thing and calls for a different procurement posture: a paid pilot on a module you select, benchmarked against a human baseline you timed.

    On integration plumbing. The Model Context Protocol is now the standard way to expose tools to an agent, and it stopped being any one vendor's asset in December 2025 when Anthropic donated it to the Agentic AI Foundation, a directed fund under the Linux Foundation, with Block and OpenAI as co-founders and Google, Microsoft, AWS and Cloudflare supporting. The latest spec revision is dated July 28, 2026. For a modernization program this matters for one practical reason: the wrappers you build around your legacy system — a read-only query tool, a job-status tool, a schema-inspection tool — should be MCP servers rather than bespoke integrations, because they will outlive whichever agent framework you pick this quarter.

    On durability.Migration runs are long. A characterization-test generation pass across a large module set, or a staged data migration, can run for hours or days, and a process that cannot resume from a crash is a process you will run twice. Durable execution engines address this directly — Temporal's official integration with the OpenAI Agents SDK reached general availability on March 23, 2026, providing crash-resume and replay. That is infrastructure, not orchestration semantics, and it is consistently underrated in agent architecture discussions.

    The Integration Constraint That Decides Most Projects

    Here is the thing that determines whether your project is feasible, and it usually gets discussed in week six instead of week one. Legacy systems rarely have APIs. An agent can read your codebase all day — reading is easy. The hard problem is the write path: how does the agent, or the system you are building around it, actually cause something to happen inside a system that was designed before REST existed?

    There are six realistic answers and they differ enormously in reliability. Pick before you scope, because the choice cascades into every other decision.

    Write pathReliabilityWhen it is rightWhat breaks
    Database or file-level integration you buildHigh — deterministic, testable, versionableThe legacy system's data model is reachable and its invariants are documented or discoverableBypasses business logic embedded in the application layer. You may enforce constraints the app enforced, or corrupt state.
    Thin service wrapper in front of the legacy systemHigh — you own the contract and can test itYou can deploy code adjacent to the legacy system and it has a callable entry pointRequires access to the legacy runtime and someone who understands its transaction semantics.
    Terminal or screen-scraping automationMedium — deterministic but brittleThe interface is frozen and change-controlledEvery UI change breaks it silently. Needs monitoring that detects drift, not just failure.
    Computer-use agent driving the UILow to medium — depends entirely on horizon and interface stabilityShort, structured, single-system flows where a wrong action is recoverableLong-horizon tasks degrade badly. Per-action latency compounds over a hundred-step session, and benchmarks measure happy paths only.
    Human-in-the-loop task queueHigh — the human is the write pathAnything irreversible, financially material, or governed by a system of recordThroughput is bounded by reviewers. That is a feature when the alternative is an unreviewed write to a ledger.
    Robotic process automation you already ownMedium — mature but rigidThe process is stable and already automatedRPA rebranded as an AI agent is the most common form of agent washing. Ask what actually changed.

    Write-path options for agents interacting with a legacy system, ranked by reliability — Frenchy Digital, 2026.

    On computer-use agents, honestly. The pitch is that if the system has a UI, an agent can drive it, and no integration work is needed. That is true in a limited and well-characterized set of conditions: structured, single-site, short-horizon flows against a stable interface, where a wrong action is recoverable. Outside those conditions it degrades badly — long-horizon multi-step tasks, drifting selectors and layouts, session interruptions, authentication walls, and anything where a wrong click cannot be undone.

    The commercial record is the most legible evidence available on this, and it is not encouraging. Both flagship consumer browser agents have been shut down. OpenAI deprecated Operator and shut it down on August 31, 2025, folding the capability into ChatGPT agent. Google shut down Project Mariner on May 4, 2026, absorbing it into Gemini Agent and AI Mode, and the stated reasons were specific and technical: heavy compute for real-time visual processing, slow performance, and form-selection errors. Two of the best-resourced teams in the industry took their consumer browser agents off the market. That is a data point about the current reliability of the capability, not about ambition.

    The capability itself is real and available — Anthropic offers computer use as an API capability where you run your own sandboxed virtual machine, alongside a macOS research preview launched March 24, 2026. Benchmark context helps calibrate: on OSWorld, 369 real operating-system tasks, the human baseline is 72.36%. Self-reported leaderboard entries for frontier models now sit in the mid-eighties, but the same models' verified scores on Anthropic's own model cards land around 72–73% on OSWorld-Verified — a six-to-eleven-point gap on identical models, where the scaffold and attempt budget explain more than the model does. And these benchmarks measure happy paths. Per-action latency compounds: a hundred-step session accumulates a lot of round trips, and each one is an opportunity to click the wrong thing.

    The architectural point people resist: a human-in-the-loop task queue is frequently the correct design, not a failure to automate. The agent does the expensive part — locating the record, assembling context, drafting the change, showing its reasoning — and a person commits it. You get most of the time savings, you get an audit trail with a named human on every write, and you get a system whose worst-case failure is a queued task nobody actioned rather than a corrupted ledger. Throughput bounded by reviewers is a real cost. It is a smaller cost than an unreviewed write to a system of record.

    Reference Architecture for a Legacy Modernization Agent

    This is the layered architecture we deploy. It is designed around one assumption: the agent will be wrong, repeatedly, in ways that are not obvious. Every layer exists to make that survivable.

    LayerWhat it is
    1. Read-only comprehension planeThe agent's default state. Repository access, static analysis output, ticket history, and runbooks — all read-only, no credentials that can mutate anything.
    2. Deterministic scaffoldCall graphs, dependency inventories, schema dumps, and coverage reports generated by conventional tooling and fed to the agent as facts rather than asked of the model.
    3. Retrieval scoped to the taskJust-in-time retrieval over preloading the whole repository. Context degrades as it grows well before the window fills — an attention budget, not a storage budget.
    4. Sandboxed executionThe agent builds and runs tests in an ephemeral container with no network egress beyond an allowlist and no production credentials, ever.
    5. Test-first change gateNo change is proposed against a module without characterization test coverage for that module. Coverage is a precondition, not a follow-up ticket.
    6. Proposal, not commitThe agent opens a change proposal with the diff, the tests it ran, the tests it added, and its stated assumptions. A human merges.
    7. Approval gates that resolve the targetAny action against a system of record displays the fully resolved target — the exact table, record, environment, and account — not a templated description of it.
    8. Strangler facadeA routing layer in front of the legacy system that lets you move one capability at a time and roll it back in one step.
    9. Audit log of every tool callActor, tool, arguments, resolved target, environment, model and prompt version, outcome, and the human decision. Append-only.
    10. Private eval in CIReal tickets, isolated environments, outcome grading. Runs on every prompt, tool, or model change and on every model upgrade.

    Reference architecture for an AI agent operating against a legacy system — Frenchy Digital, 2026.

    Two layers carry most of the weight. Layer 2, the deterministic scaffold, is the one teams skip and the one that most improves results. Call graphs, dependency inventories, schema dumps, and coverage reports are cheap to produce with conventional tooling, they are reproducible, and they are correct. Feeding them to the agent as facts is strictly better than asking the model to infer them, and it collapses a large amount of token spend and hallucination surface at the same time. Run the deterministic tools first, always.

    Layer 5, the test-first change gate, is the one that gets deferred and should not be. The rule we enforce is simple: no change is proposed against a module that lacks characterization coverage. Not “we will add tests after” — no proposal at all. This is unpopular in week two and it is the reason these programs survive month six.

    A note on layer 6. The distinction between a proposal and a commit is the whole safety model, and it degrades quietly. Teams start with careful review of every diff, then reviewers get faster, then approvals become clicks. Instrument it: track review duration as a distribution rather than an average, track the share of proposals modified before merge, and treat a flat modification rate as a broken control rather than a perfect model. A review step nobody performs is a review step you have removed while keeping the paperwork.

    Risk and Governance for an Agent Touching a System of Record

    An agent operating against a system of record is a high-blast-radius agent and needs the controls that go with that. The specific hazard in legacy work is that the systems involved usually lack the fine-grained authorization that would contain a mistake. There is often no per-record permission model to fall back on. The credential either can update the ledger or it cannot.

    ControlWhy it matters here specificallyImplementation
    Least privilege by defaultAn agent with production write credentials has the blast radius of an unsupervised contractor with root. Legacy systems rarely have fine-grained authorization to fall back on.Read-only tokens as the default. Write scopes granted per workflow, time-boxed, and separately audited.
    Sandboxed executionGenerated code and generated tests both execute. A build script is arbitrary code execution by another name.Ephemeral containers, no production network reachability, egress allowlist, no long-lived secrets in the environment.
    Approval gates showing the resolved targetAn approval that says 'update customer record' is not an approval. The human must see which record, in which environment.Render the fully resolved action — environment, connection, table, key, and the diff — at the moment of approval.
    Prompt injection as a codebase propertyA legacy repository is a large corpus of untrusted text: comments, string literals, fixtures, vendored code, ticket bodies. Any of it can read as instruction.Treat all repository content as data. Allowlist tools, deny arguments the human never supplied, and assume injection will eventually succeed — design for blast radius, not prevention.
    Audit logging of tool callsWhen something goes wrong in a system of record, the question is what the system did and on whose authority. A transcript of model output does not answer that.Append-only log with actor, tool, arguments, resolved target, model version, and human decision. Retained longer than the data it describes.
    Model and prompt version pinningBehavior can change under a floating alias with no commit in your repository. That defeats change control entirely.Pin explicit versions. Keep prompts and tool definitions in the repository under review. Canary model upgrades like schema migrations.
    Cost and runaway controlsAgents use roughly 4× the tokens of a chat interaction and multi-agent systems roughly 15×, per Anthropic's own engineering write-up. A migration loop can burn a budget overnight.Hard per-session budgets, per-workflow cost caps, and alerts on anomalous token growth. Advisory ceilings the model can see are not hard caps.
    Human accountability for every mergeAttribution matters when a regression reaches a system of record. 'The agent did it' is not an incident report.Every change carries a named human reviewer. No auto-merge on any path that touches the legacy system.

    Governance controls for AI agents operating against legacy systems of record. Cross-reference the OWASP Top 10 for LLM Applications and the NIST AI Risk Management Framework.

    Prompt injection deserves a paragraph of its own, because the legacy context makes it worse. A codebase is a large body of text that your organization did not fully author and has not fully read. Comments written by contractors who left in 2003. String literals containing customer-supplied data. Test fixtures built from production exports. Vendored dependencies nobody audited. README files, migration scripts, and the ticket bodies your agent reads to understand a change request. Any of it can contain text that a model interprets as instruction rather than data.

    Prompt injection is not solved, and nothing in this article makes an agent safe.There is no filter, system prompt, or guardrail model that reliably separates instruction from data in a corpus you do not control. The only defensible posture is defense-in-depth and blast-radius reduction: assume injection will eventually succeed, and design so that when it does, the worst available outcome is a rejected proposal rather than a production write. That means read-only by default, no production credentials in the agent's environment, tool allowlists per workflow, rejection of tool arguments the human never supplied, and approval gates that display the fully resolved target of every action.

    The approval-gate detail is worth stating precisely because it is where well-intentioned designs fail. An approval prompt that reads “update the customer record” is not an approval — it is a template. The human must see the resolved values: which environment, which connection, which table, which key, and the actual diff. If an injection has redirected the target, the resolved display is the layer where a human has any chance of noticing.

    On cost controls. Modernization workloads are long-running and read-heavy, which is exactly the shape that produces surprise invoices. Anthropic's figures give the scale: agents use roughly 4× the tokens of a chat interaction, multi-agent systems roughly 15×. As of August 2026 the mechanics that matter most on Anthropic's published pricing are prompt caching — a five-minute cache write costs 1.25× base input, a one-hour write 2×, and a cache read 0.1×, so the break-even is one read for the five-minute TTL and two for the one-hour — and the Batch API at 50% off both input and output, which stacks with caching. Both are highly applicable here, because a codebase-comprehension prompt has an enormous stable prefix and most archaeology work is not latency-sensitive.

    Two traps worth knowing before you build a budget. The minimum cacheable prefix is model-dependent and non-monotonic — 512 tokens on the newest tier, 1,024 or 2,048 on others, and 4,096 on several — and below the floor the request silently does not cache at all. And the tokenizer changed: Claude 4.7 and later, including Sonnet 5, produce roughly 30% more tokens for the same text. The per-token price did not change; the cost per request did. Re-baseline with a token-counting call rather than applying a multiplier to last quarter's numbers.

    Evaluation: Why Vendor Coding Benchmarks Tell You Nothing About Your Codebase

    You cannot manage this work without measurement, and the measurement you are being offered is not the measurement you need. Public coding benchmarks have a contamination problem severe enough that leaderboard position carries almost no information about performance on a codebase the model has never seen — which is precisely your situation.

    The clearest single comparison: on SWE-rebench's fresh, decontaminated task set — 111 problems drawn from 65 repositories, collected between May 15 and July 1, 2026 — frontier models land at roughly 64%. The figures circulating from vendor self-reports on SWE-bench Verified are around 95%. Same class of task, roughly 30 points apart, and the main variable is whether the model could plausibly have encountered the problem during training.

    Benchmark or findingThe numberWhat it actually tells you
    SWE-bench Verified (official board)79.2% — live-SWE-agent with Claude Opus 4.5, December 15, 2025The board has accepted only academic and research submissions with open methods since November 18, 2025, and holds 134 submissions. No 2026 frontier entries exist. Circulating 95–96% figures are vendor self-reports, not audited.
    SWE-rebench (fresh, decontaminated)~64% — 111 problems across 65 repositories, collected May 15 to July 1, 2026The closest available proxy for unseen code. Roughly 30 points below the vendor figures on the contaminated public set. Cost per problem varied by roughly 3× across models scoring within a point of each other.
    OpenAI's SWE-bench Verified audit59.4% of 138 audited failed problems had material test or description flawsOpenAI formally retired the benchmark in February 2026. 35.5% had overly narrow tests enforcing implementation details; 18.8% had wide tests checking unspecified functionality.
    Contamination measurementModels score 3× better on SWE-bench Verified than on decontaminated equivalents, and 6× better at locating edited files without project contextThe paper describes the file-location result as logically impossible without prior exposure. This is the clearest signal that public benchmark position is not transferable.
    Reward hacking across agent benchmarksUC Berkeley researchers reached 100% on Terminal-Bench, SWE-bench Verified and SWE-bench Pro without solving a single taskVia pytest conftest hooks, config leakage, prompt-injected judges, and VM state manipulation. Follow-up work found 219 distinct flaws across ten benchmarks.
    OSWorld (computer use)Human baseline 72.36% across 369 real-OS tasksSelf-reported leaderboard entries run 6 to 11 points above the same models' verified scores on Anthropic's own model cards. The scaffold and attempt budget explain more than the model does.
    Cost-controlled evaluationPrinceton's HAL work found agents can be 100× more expensive while only 1% betterAccuracy-only leaderboards hide the economics. Higher reasoning effort reduced accuracy in the majority of runs in HAL's 21,730-rollout study.

    Coding-benchmark standings and their validity problems, with primary sources — as of August 2026.

    Three findings make the case that this is a structural problem rather than a quirk of one benchmark. First, OpenAI formally retired SWE-bench Verified in February 2026 after auditing 138 problems its models failed, each reviewed by at least six engineers: 59.4% had material test or description flaws. Second, contamination research found models score three times better on SWE-bench Verified than on decontaminated equivalents and six times better at locating edited files without project context, a result the authors describe as logically impossible without prior exposure. Third, UC Berkeley researchers reached 100% on several major agent benchmarks without solving a single task, using pytest hooks, config leakage, and prompt-injected judges.

    The practical conclusion for a modernization program is short: build a private eval from your own tickets. Anthropic's engineering guidance is the best primary source on the method, and its core advice transfers cleanly. Start with 20 to 50 tasks taken from real failures. Run each trial isolated from a clean environment, because shared state produces correlated failures and inflated scores. Grade the outcome rather than the path — rigid tool-sequence matching penalizes an agent for solving the problem a different way. Calibrate any LLM judge against human labels, and read sampled transcripts weekly rather than trusting the aggregate. Run the whole thing in CI on every agent change and every model upgrade.

    • Weight the set toward legacy failure classes: Silent behavioral drift, boundary conditions, dead code that turns out to be live, undocumented data assumptions, and encoding or precision issues. These are the failures that reach production in an old system.
    • Measure reliability, not just success: Sierra's τ-bench work established the distinction: pass^k requires all k trials to succeed. A model with pass^1 under 50% can fall to roughly 25% at pass^8 — a reliability collapse completely invisible to a single-attempt metric. On a system of record you care about the eighth attempt.
    • Measure cost alongside accuracy: Princeton's HAL research found agents can be 100× more expensive while only 1% better, and that higher reasoning effort reduced accuracy in the majority of runs across 21,730 rollouts. Accuracy-only comparison will lead you to an expensive configuration that is not better.
    • Sample production traces continuously: LangChain's guidance suggests starting at 10% trace sampling, with at least 20 labeled examples to calibrate a judge; human reviewers sustain roughly 50 to 100 traces per hour. Budget the human time explicitly — it is the part that gets cut.
    • A 100%-pass eval has stopped telling you anything: It tracks regressions and gives no improvement signal. When your set saturates, add harder tickets from the same backlog.

    One more caution on model selection. Benchmark scores also vary enormously with the harness rather than the model — bare-model, vendor-scaffolded, and full-system leaderboard variants of the same benchmark can differ by 30 to 50 points. If a vendor quotes you a number, the harness matters as much as the model name, and neither predicts your codebase.

    The Sequencing Plan: Comprehension, Tests, Increments — Never a Rewrite

    Everything above resolves into an order of operations. The order is not negotiable, and most failed modernization programs failed because they ran it backwards — starting with a target architecture and a cutover date, then discovering in month nine what the system actually does.

    PhaseDurationWhat happensExit criteria
    Phase 0 — Baseline1–2 weeksKnow what you have and how it behaves today: inventory, static analysis, ticket history, current change lead time, current defect rate.A written inventory and a measured baseline. If you cannot state today's change lead time, you cannot claim an improvement later.
    Phase 1 — Comprehension3–6 weeksDependency map, module documentation, business-rule extraction into reviewable specifications, identification of dead versus live code.A specification document a domain expert has signed off on, and a dependency map validated against production telemetry.
    Phase 2 — Safety net4–10 weeksCharacterization tests over the modules in scope, plus the CI infrastructure to run them. Agent-generated, human-curated.Coverage over the seam you intend to move first, and a green build that fails loudly when behavior changes.
    Phase 3 — First seam4–8 weeksOne capability moved behind a facade, traffic shifted incrementally, rollback tested. Deliberately small.The seam is live, revertible in one step, and the team has done a cutover once under real conditions.
    Phase 4 — Repeat and retireOngoingAdditional seams on a cadence. The legacy footprint shrinks; each step ships value.A retirement schedule with real dates, driven by seams completed rather than by a program milestone.
    Never — Big-bang rewriteReproduce every undocumented behavior of a system nobody fully understands, on a deadline, while the original keeps changing.All risk lands on one cutover date, no value ships until then, and there is no rollback. Adding AI does not change the arithmetic.

    Modernization sequencing with exit criteria per phase — Frenchy Digital, 2026.

    On the strangler fig pattern. The name comes from Martin Fowler's description of the strangler fig, which grows around a host tree and gradually replaces it. Applied to software: you build the new system around the edges of the old one, route traffic capability by capability, and retire the original piece by piece. It predates the current wave of AI tooling by two decades, and that is a point in its favor — it is the migration approach with the longest track record of not producing catastrophes.

    What AI changes is the cost of the phases, not the order. Comprehension gets meaningfully cheaper. Characterization test generation gets meaningfully cheaper. Seam selection, cutover decisions, and business-rule validation do not — those are judgment, and judgment on a system whose behavior is undocumented remains the expensive scarce input. A program that budgets for cheaper comprehension and the same amount of judgment is budgeted correctly.

    Phase 0 is the one that gets skipped and the one you will most regret skipping. If you do not measure current change lead time, current defect rate, and current time-to-onboard a new engineer onto the system, you will have no way to demonstrate improvement — and given what METR measured about perception, you will not be able to feel it accurately either. Baseline before you start. It costs a week.

    On big-bang rewrites, one more consideration beyond the usual argument. A rewrite requires the organization to freeze the original while the replacement is built, which never happens, so the target keeps moving. It concentrates all risk on a single cutover event with no rollback. And it delivers zero value until the end, which means it is the first thing cut when budgets tighten in month eleven — leaving you with a half-built replacement and the original system still running. The exceptions are genuinely small systems and systems being replaced by a commercial product rather than rewritten.

    Cost Bands, Timelines, and What Actually Drives Them

    These are the bands Frenchy Digital uses to scope AI agent engagements in 2026, applied to legacy modernization work. They assume the comprehension and test phases are in scope rather than assumed away, because assuming them away is what makes these programs overrun.

    EngagementRangeTimelineTypical scope
    Discovery + architecture review$9k–$22k2–4 weeksSystem inventory, dependency map, static-analysis baseline, test-coverage assessment, candidate first seam, written modernization sequence
    Single production agent (one workflow, evals, observability)$30k–$80k5–10 weeksOne workflow end to end — typically comprehension or test generation — with a private eval set, tracing, cost controls, and audit logging
    Multi-workflow agent platform with integrations$80k–$200k10–18 weeksSeveral workflows, the strangler facade, integration to the legacy write path, CI evals, human review queue, runbooks
    Enterprise / regulated build (SOC 2 posture, HITL, audit logging)$200k–$450k+16–26 weeksMulti-team rollout, segregated environments, full approval-gate and audit pipeline, DR testing, documentation package

    Frenchy Digital cost bands for AI agent engagements applied to legacy modernization, 2026.

    Senior-led delivery runs $150 to $225 per hour, and ongoing retainers run $2,500 to $9,500 per month covering model and dependency upgrades, eval expansion, incident response, and a quarterly technical review. Every engagement carries a 30-day post-launch warranty, and you receive a written scope with a fixed-price phased proposal within 5 business days of the discovery call.

    What moves a project within a band, in rough order of impact: how much documentation survives; whether anyone who worked on the original system is still reachable; whether production telemetry exists that can distinguish live code from dead; whether the system has any programmatic write path at all; how many downstream consumers depend on undocumented behavior; and whether a domain expert is genuinely allocated rather than nominally assigned. The last one is the most common cause of overrun and the least often budgeted.

    Included at every tier: the system inventory and dependency map, a deterministic static-analysis baseline, characterization test coverage over the modules in scope, a private eval set built from your own tickets, audit logging of every agent tool call, approval gates on any write path, and full source-code and IP ownership transferred to you at delivery. Frenchy Digital is a senior-led Black-owned Los Angeles agency and we do not build lock-in.

    One budgeting note that consistently surprises people: the model spend is not the cost driver. Even on a large comprehension pass, inference is typically a small fraction of a phase budget — the engineering time to review, correct, and integrate the output is the dominant line, and it is roughly the same whether the model is cheap or expensive. That is an argument for choosing the model that produces the least rework rather than the lowest per-token price, and for spending the caching and batching effort described earlier, which reduces the small line without touching the large one.

    Red Flags in a Legacy Modernization Proposal

    Every one of these has appeared in a real proposal we have been asked to review. None are hypothetical.

    Red flagWhy it matters
    A specific migration speedup percentage with no methodologyNo independently measured productivity figure exists for this work. A percentage without an N, a baseline, and a named measurer is marketing.
    A fully automated end-to-end rewrite proposalIt requires the tool to be right about behavior nobody has documented, and it gives you no incremental checkpoint at which to discover it was not.
    No characterization tests before the first changeWithout a behavioral baseline you cannot tell a successful migration from a silent regression. This is the single most common reason these programs fail.
    Agent given production write credentialsThe blast radius is the whole system of record, and legacy systems rarely have the fine-grained authorization to contain it.
    Benchmark scores presented as evidence about your codebasePublic coding benchmarks are contaminated and reward-hackable. Frontier scores drop roughly 30 points on fresh decontaminated tasks.
    A big-bang cutover date in the first proposalA date set before comprehension is complete is a date set before anyone knows what the work is.
    No domain expert allocated to the programThe value is in the business rules, and only a domain expert can tell you which extracted rule is correct and which is a forty-year-old bug.
    'Agentic' applied to an existing RPA or scripted pipelineGartner estimates only around 130 of the thousands of self-described agentic AI vendors are genuinely agentic. Ask precisely what the model decides.
    No eval set built from your own ticketsWithout one, every model upgrade is an unmeasured change to a system touching your system of record.
    Vendor unwilling to run a paid pilot on a module you chooseYou choosing the module is the entire point. A vendor-selected demo module tells you what the tool does well, not what it will do on your code.

    The Frenchy Digital red-flag list for AI-assisted legacy modernization proposals, 2026.

    Ask any vendor for one thing before you sign: a paid pilot on a module you select, measured against a baseline you timed, with the characterization tests written first. A vendor who will not do that is selling a demo. A vendor who will is showing you what the tooling actually does on your code, which is the only evidence that exists.

    Frenchy Digital buyer’s principle

    Limitations and Honest Failure Modes

    This article argues for a specific approach, so here are the ways that approach fails and the parts of the argument that are weakest.

    • The evidence gap cuts both ways: There is no measured productivity number supporting AI-assisted modernization, and there is also no measured number refuting it. The honest position is uncertainty, not pessimism. METR is the closest thing to a rigorous measurement in adjacent territory, and METR itself says it is historical.
    • Comprehension output can be confidently wrong: A generated dependency map or business-rule extraction that is 95% correct is dangerous precisely because it is mostly right — reviewers calibrate to its accuracy and stop checking. Every comprehension artifact needs validation against something external: production telemetry, a domain expert, or a test that actually runs.
    • Characterization tests pin bugs as well as features: That is the intent, and it is also a trap. A test suite that enshrines a forty-year-old defect makes fixing that defect look like a regression. Tag tests as contract versus incidental as you go, or you will spend the migration arguing about which failures are real.
    • Prompt injection has no solution: Every control in the governance section reduces blast radius. None of them prevent injection. If your risk tolerance requires prevention rather than containment, do not give an agent a write path to the system of record at all.
    • Private evals are expensive and go stale: Building 20 to 50 real tasks with clean environments is real engineering work, and human review time is the part that gets cut first. An eval nobody maintains is worse than none, because it produces a green signal that means nothing.
    • Human-in-the-loop degrades silently: Review quality decays as volume rises and reviewers acclimate. Without instrumentation on review duration and modification rate, you will not notice until an incident tells you.
    • The organizational constraint usually dominates the technical one: Gartner's April 2026 survey found skills gaps and data quality each cited by 38% of I&O leaders as the blocker — ahead of tooling. Our own experience matches: the programs that stall usually stall because the domain expert was never really available, not because the model was not good enough.
    • Model behavior drifts under you: Vendors ship new versions continuously and behavior changes without a commit in your repository. Pin versions, canary upgrades, and keep the eval set running — otherwise the answer to what your system did last quarter is a guess.

    None of this argues against doing the work. It argues for doing it in an order where being wrong is cheap, with instruments that tell you the truth rather than what you hoped. The organizations getting real value from AI agents on legacy systems are, without exception in our experience, the ones that measured the before-state and kept measuring.

    And the framing that matters most for a CTO deciding whether to start: the case for modernization does not depend on AI being transformative. A system nobody understands is a business risk regardless of tooling. Comprehension, documentation, and a test suite are worth building whether or not a model helped write them — and if the tooling turns out to be less capable than claimed, you still end up with an understood system instead of an untouchable one. Sequence the program so that it delivers that outcome either way.

    Where to Go Next

    Legacy modernization sits at the intersection of several harder problems, and the rest of this cluster covers them in depth. If your comprehension layer needs to serve answers across a large corpus, start with RAG for enterprise knowledge bases. If you are deciding how many agents the archaeology phase should fan out to, multi-agent systems architecture covers both sides of that argument with the primary sources.

    For the private eval set and the tracing that makes it usable, see AI agent evaluation and observability. For the injection surface a legacy codebase represents, see prompt injection and the OWASP LLM Top 10. And for the caching, batching, and token mechanics that determine what a long comprehension pass actually costs, see LLM cost optimization and token economics.

    Sitting on a System Nobody Wants to Touch?

    Book a free 60-minute discovery call with Frenchy Digital — a senior-led Black-owned LA agency. You leave with a comprehension plan, a test-coverage baseline, a candidate first seam, and a fixed-price phased proposal within 5 business days. Call +1 (424) 272-5601.

    Sitting on a System Nobody Wants to Touch?

    Book a free 60-minute discovery call. You leave with a comprehension plan, a test-coverage baseline, and a fixed-price phased proposal within 5 business days.

    1517 S Bentley Ave Unit 204, Los Angeles CA 90025

    Frequently Asked Questions

    Sources & References

    1. 1METR — Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
    2. 2METR — Uplift Study Update (Feb 24, 2026)
    3. 3arXiv 2507.09089 — METR developer productivity RCT (paper)
    4. 4IBM — Agentic AI for Smarter Mainframe Modernization with watsonx Code Assistant for Z
    5. 5Martin Fowler — Strangler Fig Application
    6. 6SWE-rebench — fresh, decontaminated SWE task leaderboard
    7. 7OpenAI — Why We No Longer Evaluate SWE-bench Verified (Feb 2026)
    8. 8arXiv 2512.10218 — Benchmark contamination in coding evaluation
    9. 9UC Berkeley RDI — Trustworthy Benchmarks (reward hacking across agent benchmarks)
    10. 10Anthropic — Demystifying Evals for AI Agents (Jan 2026)
    11. 11Anthropic — Effective Context Engineering for AI Agents
    12. 12Anthropic — How We Built Our Multi-Agent Research System
    13. 13Anthropic — Computer Use Tool Documentation
    14. 14Anthropic — Model Pricing (verified August 2026)
    15. 15OSWorld — Benchmarking Multimodal Agents in Real Computer Environments
    16. 16OpenAI — Introducing ChatGPT Agent (Operator successor)
    17. 17Android Authority — Google Shuts Down Project Mariner (May 2026)
    18. 18Gartner — AI Projects in I&O Stall Ahead of Meaningful ROI Returns (Apr 7, 2026)
    19. 19Stanford HAI — 2026 AI Index Report
    20. 20Deloitte — State of AI in the Enterprise 2026
    21. 21OWASP — Top 10 for Large Language Model Applications
    22. 22NIST — AI Risk Management Framework
    23. 23Anthropic — Donating MCP to the Agentic AI Foundation
    24. 24Princeton HAL — Holistic Agent Leaderboard (cost-controlled evaluation)
    Chris Machetto - CEO & Founder of Frenchy Digital

    Chris Machetto

    CEO & Founder of Frenchy Digital. Building apps and digital products since 2019 for startups and enterprises across LA, San Francisco, Paris, Geneva, and more globally.