When Is Machine Learning Worth the Investment? Lessons from Real-World Development Projects

Some of the most useful advice we give clients is three words long. Don't build it yet.

A typical version of that call goes like this. A managing director has a board meeting on Thursday, a competitor has put "AI-powered" in a press release, and there's money set aside. We ask four plain questions about data, decisions and costs, and by the third the line goes quiet. It happens often enough to call it a rule. Machine learning pays off when four conditions hold at once, and most failed projects skipped checking at least one of them.

None of this makes me a sceptic. I spend my working week on these systems, and I've seen them pay for themselves many times over. I've also seen companies spend six figures finding out something an afternoon with a spreadsheet would have told them. The difference was never the algorithm. It was whether anyone checked the conditions before the contract was signed.

Most machine learning money is lost before anyone writes a line of code

The failure numbers are bad enough to make anyone careful. A RAND Corporation report published in August 2024 found that, by some estimates, more than 80% of AI projects fail, roughly twice the rate of IT projects that don't involve AI. The root causes its researchers listed are telling. Teams misunderstood the problem. They lacked the data to train a model. They chased the newest technology instead of a real business problem. Not one of those is a coding failure.

Gartner points the same way. In February 2025, it predicted that through 2026 organisations will abandon 60% of AI projects unsupported by AI-ready data, after a survey of 1,203 data management leaders found that 63% either lacked the right data practices or weren't sure whether they had them. The return side is thin as well. In McKinsey's State of AI survey, published in August 2026, nearly nine in ten respondents said their organisations use AI regularly in at least one business function, yet only 37% said it had contributed positively to EBIT.

Every company needs an AI strategy now.

You hear that line at every business breakfast and trade show going. It's half right. Every company needs a view on AI, the same way it needs a view on cyber security. A view is not a model, though, and a strategy slide is not a business case. The Office for National Statistics reported in July 2026 that the share of UK businesses with 10 or more employees using at least one AI technology rose from around 12% in late 2023 to around 35% by June 2026. Only 10% of those adopters describe their use as extensive. Adoption has nearly tripled. Depth has barely moved. My reading is that plenty of firms bought a licence and a strategy deck, and far fewer found a decision worth automating. That last part is my inference, not the report's.

That gap between using machine learning and earning from it is where we spend most of our time. When firms come to us for machine learning development services, the first job is rarely modelling. It's checking whether the business case survives contact with the data. We call that check the four-test rule. A project has to pass the data test, the frequency test, the cost-of-error test and the baseline test, all at the same time. Three out of four isn't a near miss. It's a project that quietly underdelivers for eighteen months before someone in finance cancels it.

The data test: if the history isn't there, no budget will put it there

Every model learns from the past, so the past has to exist in a usable form. Two things matter. You need enough history to show the patterns you care about, which for anything seasonal means two or three full years at the very least. And you need a clean target, meaning the thing you want to predict was recorded consistently, in one place, with the same meaning over time.

The target is where we see most projects wobble. Returns logged as negative sales one year and as a separate code the next. A product range renamed during a system migration, so half the history sits under codes nobody remembers. Branch data typed into Excel on Monday mornings by whoever was in first. None of it is fatal. It just has to be found before the budget is agreed.

Denys Topalenko, a Machine Learning Engineer on our team, runs this check on almost every new enquiry. "I ask for the raw export, not the dashboard, and then I look at three things: how many years of history there really are, where the gaps fall, and how the number we want to predict was actually recorded, because 'sales' in one system can mean orders and in another it means dispatches. After that I build the simplest baseline I can, usually the same week last year. If a model can't clearly beat that on months it hasn't seen, I'd rather tell the client now than six months in. It's a much shorter conversation."

• Passes when: you hold two to three years or more of consistent records, and the outcome you want to predict sits in one field with one meaning.

• Fails when: the history lives in spreadsheets that changed format every year, or the target has to be rebuilt by hand before anyone can use it.

The frequency test: a decision you make once a year can't pay back a model

A model is a machine for making the same decision many times, a little better each time. The returns come from repetition. Setting reorder quantities for 4,000 product lines every week qualifies. Choosing a new warehouse site once a decade doesn't, however much money rides on it. That one deserves a good analyst and a strong coffee.

A small improvement multiplied across thousands of decisions compounds into real money. A large improvement on one decision is a consultancy project with extra steps. We've had more than one request to "predict" next year's overall budget, and a model trained on a handful of annual data points will do no better than a careful finance director. It will just cost more.

• Passes when: the decision is made daily or weekly, across many items, customers or locations, and the output feeds straight into a process.

• Fails when: it happens a handful of times a year, or a manager has to sign off every single prediction anyway.

The cost-of-error test: a prediction is worth the gap between a right call and a wrong one

Accuracy figures mean nothing until you price them. Ask finance two questions. What does a wrong prediction cost us? What does a right one save? If the gap between those two numbers is small, even an excellent model earns very little.

Take an illustrative case, with made-up figures rather than a client's. A food wholesaler overstocks a chilled line and writes off £3,000 of product, or understocks and loses £5,000 of orders to a rival down the road. Those mistakes are expensive and frequent, so a model that cuts them even modestly pays back quickly. Now picture a model that predicts which newsletter subscribers will open next week's email. Right or wrong, the cost is pennies. Same technology. Very different business case.

Errors are rarely symmetrical, either. Running out of a best-seller hurts more than holding a few extra boxes of a slow line. A good project builds that imbalance in from the first week. A bad one reports a single accuracy number and hopes nobody asks which way the errors fall.

• Passes when: you can put a pound figure on both a wrong call and a right call, and the gap is large enough to matter at your volume.

• Fails when: nobody can price an error, or a wrong prediction is spotted and reversed at almost no cost.

The baseline test: if a simple rule gets you most of the way, keep the simple rule

Before any model, build the boring version. Last year's figure for the same week. A twelve-week moving average. The rule your most experienced planner already applies in her head. Then measure it properly, on recent data it hasn't seen.

This step does two jobs. It shows how much room a model has to improve on what you do today. And it gives everyone an honest yardstick, so nobody can claim a model is "85% accurate" without saying compared to what. We've seen baselines win. When they do, we say so, and the client keeps the money.

Here's the uncomfortable part. A team that skips the baseline can't tell whether its model is good. It can only tell that it's expensive.

• Passes when: a model beats your current rule by a margin that's still worth having after hosting, retraining and staff time.

• Fails when: there's no baseline at all, or the improvement disappears once you count what it costs to keep the model running.

The four tests fit on one page, and checking them costs almost nothing

Here is the whole rule in one place. None of the cheap checks needs a data scientist. Most need a finance lead, an operations manager and a quiet afternoon.

Test	Passes when	Fails when	Cheap way to check
Data	Two to three years or more of consistent records, one clear target field	Gaps, format changes, target rebuilt by hand	Export the raw table and chart the target month by month
Frequency	The decision repeats daily or weekly across many items or sites	A few decisions a year, or every output needs manual sign-off	Count how many times the decision was made last year
Cost of error	A wrong call and a right call can both be priced in pounds	Errors are cheap or instantly reversible	Ask finance what last year's three worst misses cost
Baseline	A model beats a simple rule by a margin worth paying for	No baseline, or the gain vanishes after running costs	Build the rule in Excel and test it on last quarter

A fail on any row isn't the end of the idea. It's a to-do list.

One of the largest US retailers passed all four tests, and the results show it

Our goods demand forecasting project is the clearest worked example we have, because it passed every test before we wrote any code. The client, one of the largest retailers in the US, needed to predict demand for specific products across the country: when customers would buy, how they would behave, and how much stock each location needed to hold. Until then, planning had run on spreadsheets and gut feel.

Run it through the rule.

The data test. The retailer held historical sales data for different product types, which is exactly what a demand model trains on. The target, product-level sales, was a number the business already recorded and cared about.

The frequency test. Stock decisions happen constantly, for every product at every location. At that scale, even a small gain per decision adds up quickly.

The cost-of-error test. Overstock ties up storage that costs money every day it sits there. Understock loses the sale, and sometimes the customer. Both sides of the error had a price, and the price was high.

The baseline test. Spreadsheets and instinct are a genuine baseline, and a beatable one. The real question was whether a model could beat the planners' method by enough to matter.

It did. The system we built trains on historical sales for different product types and produces dated, product-level demand predictions, and those predictions land with 91% accuracy. After go-live, revenue improved by 34%, and storage capacity utilisation improved by 65%. Those numbers came from a project where all four conditions held at once. I'd be suspicious of anyone promising similar figures on a project that fails even one test.

The lesson for a UK firm with 40 branches rather than thousands of stores isn't about scale. It's about the order of work. Check the four conditions first, build the baseline second, and only then talk about models.

A one-page worksheet to fill in before you sign anything

Print this, hand it to your finance and operations leads, and leave a few lines of space under each question. If nobody can answer one of them yet, that's your answer for now.

1. What exactly do we want to predict, and where is it recorded today? A good answer names one field in one system, with the same meaning for at least two or three years.

2. How much usable history do we hold, and where are the gaps? A good answer gives a start date, a list of known gaps and the reason for each.

3. How often do we make this decision, and across how many items or sites? A good answer says weekly or more, across hundreds of products, customers or locations.

4. What did our three worst wrong calls cost us last year? A good answer is a pound figure from finance, not a guess from the meeting room.

5. What does a right call save us each time we make it? A good answer is a per-decision figure you can multiply by the frequency from question three.

6. What's our current rule, and how accurate is it on recent data? A good answer is a baseline written down, measured on last quarter, with an error figure attached.

7. Who will act on each prediction, and what will they do differently? A good answer names a role and a changed process, not "the team will gain insight".

8. What will it cost to run, monitor and retrain the model once it's live? A good answer is a yearly figure covering hosting, data upkeep and staff time.

Most of the time, "don't build it yet" isn't a no at all but a date in the diary.