We Already Tried That: Why AI Failures Need a Retry Ledger

Most families keep a box of recipe cards, and a few of those cards carry a note in the margin that says "didn't work." Sometimes the note means nobody liked the dish. Sometimes it means the old oven ran cold and the bread never rose. Those are two very different failures, but once the card goes to the back of the box, they look exactly the same.

"We already tried that" is something I hear from customers time and time again, and it works the same way. When an AI idea failed last year, the answer to "should we try it again?" depends on which kind of note is in the margin. If the failure came from the technology rather than the organization, the idea belongs in a kept backlog that gets retested every time the models improve, because they now improve every few months.

Key figures (November 2026)

• The length of task frontier AI agents can complete has doubled every 196.5 days since 2019, and every 88.6 days for models released since 2024 (METR, January 2026).
• OpenAI's GPT-4 scored around the top 10 percent of test takers on a simulated bar exam, where its predecessor GPT-3.5 scored in the bottom 10 percent (GPT-4 Technical Report).

Two kinds of "didn't work"

When a customer tells me they already tried something, I can often agree with them. Plenty of tools and tactics fail because they don't fit the culture. A team that runs on relationships won't live inside a rigid workflow tool, and a company that punishes mistakes won't get much from an experiment that depends on people admitting what went wrong. That is the dish nobody liked, and trying the same recipe again on the same family won't change the result.

But many failed projects never got that far. The technology didn't do what the organization thought it would, or it did it too slowly, too unreliably or at a cost that made no sense. That is the cold oven. The recipe might have been fine, and the family might have loved it, but nobody got to find out.

For most of the history of enterprise software, the two notes could share a drawer without much harm, because the oven changed slowly. A system you bought rarely got dramatically better between one budget cycle and the next, so a project that failed on technology would fail again on the same technology for years.

The oven now gets replaced every few months

AI broke that assumption. METR, a research group that measures how long a task an AI agent can finish on its own, found that this task length has doubled every 196.5 days since 2019, a little over six months, and faster for the most recent models. The jump between releases can be dramatic. In OpenAI's own technical report, GPT-4 scored around the top 10 percent on a simulated bar exam, where GPT-3.5 had landed in the bottom 10 percent.

So a pilot that failed in early 2025 was tested on an oven that has since been replaced, possibly more than once. Some of those early efforts were Brochure Mode experiments that were never going to change much. (Also read: We're Using AI Like It's 1995) Others were serious attempts that ran into what the model of the day could do. The verdict on those was accurate on the day it was reached, and it has an expiration date that nobody wrote down.

Why nobody reopens the box

If retesting is that valuable, it's fair to ask why so few companies do it. The answer sits in how organizations reward people.

The person who stopped a failed pilot usually looks prudent, and the decision becomes part of the company's memory. The person who wants to reopen it takes on all of the risk. If the retest fails again, they wasted time on something the organization already settled, and if it works, they have quietly suggested that a colleague, or a boss, called it too early. Few incentive plans pay anyone to take that bet, so "we already tried that" becomes the safest sentence in the room.

There is also no owner. The team that ran the pilot has moved on, the budget closed, and the notes live in a slide deck nobody opens. The organization remembers the conclusion and forgets the conditions, which is the card with "didn't work" in the margin and no mention of the oven.

A proposal: the Retry Ledger

What I'd propose is something I'm calling the Retry Ledger: a kept list of ideas and trials that failed on technology, each recorded with what was tried, which model and setup it used, why it failed, and what would have to be true for it to work. Ideas that failed on culture or fit get recorded too, marked as such, so nobody mistakes them for oven problems.

The ledger only works if someone owns it and something triggers it. A sensible trigger is every major model release your team actually uses, when a small group reruns the most promising entries against the original test. A retest costs far less than the original pilot, because the hard thinking was already done the first time. Model prices also keep falling, which lowers the cost of a retest even more, and my AI Price Tracker follows those prices daily. The rerun is where the earlier failure pays off.

It also needs an incentive that points the right way. If the people who revive a ledger entry get credit for the result, and the people who shelved it get credit for documenting it well, reopening the box stops being a political risk. To get started, download the free Retry Ledger template (Excel).

The ledger is an advantage your competitors don't have

In my piece on the carriage trade I described the Studebaker Test: a company survives a substitution wave to the extent it owns something the newcomer still has to pass through. (Also read: Is AI Doing to SaaS What the Car Did to Carriage Makers?) A Retry Ledger turns past failures into that kind of asset. A competitor starting fresh has to discover what doesn't work. A company with a well-kept ledger already knows exactly where the oven ran cold, and it can be first to try again the week a better one arrives.

That also changes what "we already tried that" should mean. The useful follow-up is which note is in the margin, and which model the idea was last tested on.

Key Takeaways

• An AI idea that failed because the model fell short is a dated result, and the date matters more each release.
• Failures from culture or fit and failures from technology need separate records, or the organization retires good ideas along with bad ones.
• Incentives reward the person who shelved a pilot and penalize the person who reopens it, so retesting rarely happens without an owner.
• The Retry Ledger keeps each technology failure with its model, setup and failure conditions, and reruns the best entries at each major model release.
• A well-kept record of what failed and why becomes an advantage a newcomer can't copy.

FAQ

Why do failed AI projects deserve a second try?
Many failed because the model of the day couldn't do the job, not because the idea was wrong. METR's research shows the length of task frontier agents can complete has doubled every 196.5 days since 2019, so a test run on an older model can give a different answer on a newer one.

What is the Retry Ledger?
The Retry Ledger is a proposed practice: a kept list of AI ideas and trials that failed on technology, each recorded with the model used, why it failed and what would have to change, and retested when a major new model is released. Ideas that failed on culture or fit are recorded separately.

Who should own an AI retry backlog?
Someone with a reason to reopen it. The person or small team that owns the ledger needs credit for revived ideas, and the teams that shelved a pilot need credit for documenting it well, so reopening the box isn't a political risk.

---

Sources and References

• Brochure Mode, a framework by Martin Goetzinger (My Idea Frameworks)
• The Retry Ledger, a framework by Martin Goetzinger (My Idea Frameworks)
• The Studebaker Test, a framework by Martin Goetzinger (My Idea Frameworks)
• The Retry Ledger template (Excel), a free download from MartinGoetzinger.com
• METR, metr.org
• GPT-4 Technical Report, arxiv.org

About the Author

Martin Goetzinger has spent his career in enterprise software sales, helping large organizations such as Apple, Microsoft, and Verizon connect data, insight, and action. His work focuses on transforming how businesses measure success and create customer value through technology.

Outside the enterprise world, he writes about the five forces he believes are reshaping everything: AI, blockchain, energy, personalized health, and robotics. Not from a purely technical lens, but from a human one as to how these technologies will redefine work, wealth, and well-being.

He is based in the U.S. and publishes at www.MartinGoetzinger.com.

Disclaimer

The views expressed in this article are the personal opinions of the author and are provided for informational and educational purposes only. Nothing in this article constitutes investment advice, financial advice, legal advice, or any other form of professional advice. Do not make investment or financial decisions based on the content of this article. Always consult a qualified professional before making decisions that affect your finances, business, or livelihood.