So where’s the recession then? Before the Brexit vote, 71 per cent of economists polled by Bloomberg said a vote to leave the EU would trigger negative growth for the first time since 2009. Leave campaigner Michael Gove was ridiculed for suggesting “we’d heard enough from experts”. And yet here we are, growing as normal.
It’s more than a one-off incident. Economists are simply terrible at making predictions. A huge study by Prakash Loungani of the International Monetary Fund revealed the record of economists was barely better than guess work. “The record of failure to predict recessions is virtually unblemished,” he mocks.
The psychologist Philip Tetlock looked at political forecasts throughout the 1980s and 1990s. He found a consistent pattern of wrong predictions, associated with huge world-changing events. The CIA famously failed to predict the fall of the Berlin Wall. Professor Tetlock later wrote a book, Superforecasting: The Art and Science of Prediction, about making predictions and repeated his view that “the average expert was roughly as accurate as a dart throwing chimpanzee”.
Need for accuracy
This is bad news. Businesses need accurate forecasts. A supermarket must estimate how many cabbages it will sell, or risk overstock or selling out. Airlines must be able to judge passenger no-shows, in order to maximise revenue. It’s a basic task of business.
The mission, therefore, is to know when forecasts are reliable and when not. This involves digging into the mechanics of forecasting, and identifying the glitches and vulnerabilities. And there are some horrors in there.
One man who’s paid to help companies identify their wrongdoing is Giles Pavey, chief data scientist at dunnhumby, the retail analytics company famous for inventing the Tesco Clubcard. He can rattle off umpteen forecasting errors.
“Most modelling techniques rely on different factors being independent to each other,” says Mr Pavey. “That is the biggest mistake you can make. If you think of putting a tax on sugar, you may assume that by raising the price by 10 per cent sales will fall 10 per cent. But that is not the only relationship. You may find retailers decide not to promote sugary drinks or that families decide not to give their children sugary drinks. Things either spiral up or down. Very few models include this.”
Often the data is wrong, missing or misinterpreted. Danielle Pinnington, managing director at shopper research agency Shoppercentric, says new product launches are a great way to watch this error in action. “Big data can’t tell you what it hasn’t already measured. It can’t predict the outcome of a wholly new idea because it doesn’t have relevant data on which to base the prediction,” she says, recommending bespoke research to address this.
Ms Pinnington offers another shortcoming: “This behavioural nature of much big data also means that it can’t tell you why shoppers are behaving in that way. For example, in the retail sector it won’t provide the details of the context to purchase decisions, the mindset of the shopper or their attitudes and expectations on a given purchase occasion.” The slump in VW car sales, which came immediately after the scandal of emissions fixing, is a good example.
Looking for answers in pricing, reliability data and promotions would be entirely misleading. As Ms Pinnington notes: “This leads us to the real Achilles’ heel of big data, the fact that it captures behaviour among shoppers and not those who didn’t buy or who don’t use your product or brand.” And they are just the people you need to complete the picture.
Sometimes the tiniest glitch can cause havoc. Arne Strauss, associate professor of operational research, explains: “The number of sales may not correctly represent demand for the product. For example, if the product ran out of stock in the middle of the week, the resulting weekly sales figure does not represent the demand for that product that could have materialised if it would have been available all week.”
Solutions?
OK, those are the challenges, but what are the solutions? In fact, there are some great ways to bring order from chaos.
A key message is to limit the scope of your forecasts. French rail network SNCF uses software from Qlik to optimise staffing. In a niche area such as this, forecasts can be made with high accuracy. “By viewing the peaks on age groups in certain areas, it is possible to anticipate necessary training and recruitment,” says Hervé Genty, a project manager at the railway.
Increasing data inputs will help. For example, the weather can affect sales patterns. So why not include Met Office data? It may come as a surprise that the Met Office is delighted to help corporate partners. It currently aids United Utilities to look at the relationship between weather and water demand.
Suck in all the data you can. Builders merchants Travis Perkins adopted a package from analytics company SAS to track 100,000 SKUs (stock keeping units) across its 21 distinct businesses. It then worked with a big data specialist CoreCompete to forecast the best stock levels at each location. Crunching numbers like these is hard, but doable.
It is critical to always sense-check the data input and acknowledge whether data accuracy or sample size is good enough before relying on algorithms
If necessary, use artificial intelligence (AI). A new courier app called Stuart is hoping to offer rapid delivery services for retailers in urban centres. To forecast demand it is using AI. David Saenz, UK managing director of Stuart, says: “We use historical data from all our current clients to understand patterns in terms of time of day, transport modes requested and areas in the city, which we use to accurately forecast the potential demand coming in and the associated number of couriers required. On top of this, we also need to factor in new clients coming on the platform or any expected variations.”
He warns that relying on AI 100 per cent would be foolish. “Relying too much on data without sense-checking and taking feedback from operational experience can be toxic. In this context, it is also critical to always sense-check the data input and acknowledge whether data accuracy or sample size is good enough before relying on algorithms,” Mr Saenz says.
Even the best AI is limited. He adds: “Exact timing and breadth of sudden peaks or other black swan events are extremely difficult, if not impossible, to predict.”
A gold standard is to change forecasting from broad numbers, to judgments derived by looking at individual components. Eric Fergusson, director of retail services at commerce specialist eCommera, says: “A number of retailers, particularly those with direct mail heritage, have a relatively sophisticated budgeting process which forecasts revenue based upon prior customers repurchasing.
“This is based on historic metrics of repeat purchase and response rate to campaigns. It is, however, an aggregate forecast, rather than specifically being used to state that ‘Eric Fergusson of the Barbican’, for example, is 90 per cent likely to shop in the third week of June.
“Interestingly, not as many retailers pursue this data-led forecasting methodology as you would think, with many still preferring to roll forward historic growth rates, which neglect the underlying performance of the customer base, and can lead to ‘surprise’ gaps as recruitment slows during maturity.”
Predictive analytics is a booming trade. Bluewolf’s 2016 The State of Salesforce Report found that 81 per cent of Salesforce’s software customers globally said increasing the use of predictive analytics was the most important initiative for their sales strategies.
But it’s vital to understand the limitations of this art, as well as the solutions. We can cope with errors by economists, but poor forecasting in business can be a lot more serious.
Need for accuracy
