Why start with statistical models?

Simple methods are hard to beat

In a 2018 study of 1,045 monthly series from the M3 competition, the six most accurate methods were all statistical. In M4, none of the six pure machine learning methods beat a simple combination benchmark.

M5 is the strongest case for machine learning, with LightGBM models at the top. Even so, only 7.5% of teams beat the best benchmark, an automatically chosen exponential smoothing method with bottom-up aggregation. Those that beat it did so by a good margin, yet the winner was better than that benchmark on only 58.5% of series, measured by squared error.

Accuracy has to pay for its cost

The 2018 study also found that machine learning methods need considerably more computing than statistical ones, and called the most demanding of them impractical for hundreds of thousands of inventory items. Computers have improved since, but a catalog forecast runs on every item, so the cost per item adds up.

A better forecast also does not always change the decision. In a commentary on M5, Stephan Kolassa describes a store with no stock on hand that sells three units a week and orders in packs of eight. A forecast of four is better than a forecast of six, but the store orders one pack either way.

Outside signals are where a small model earns its place

Some demand moves for reasons a sales history does not contain. At a large German online fashion retailer, adding weather data cut forecast errors by 8.6% to 12.2% on average, and by up to 50.6% on summer weekends. That is the job of a small expert: one signal, added where it changes the outcome.

Routing has evidence behind it too. FFORMA, the runner-up in M4, used features of each series to weight seven statistical methods and one machine learning method, and its authors report that it beat simple combinations. Our design is not the same, but FFORMA shows that choosing methods by what a series looks like can work.

The layer stays thin for a reason. A study of more than 60,000 forecasts at four supply-chain companies found that larger manual adjustments to statistical forecasts generally helped, while smaller ones often damaged accuracy. Those adjustments came from people, not models, so this is only an analogy, but it is the risk we design around: adjust when the signal is strong, and otherwise leave the forecast alone.

People have to be able to question it

A forecast nobody trusts does not get used. Kolassa argues that forecasting tools must be explainable, both to be debugged and to earn the trust of the people using them, or they will simply be ignored. In incentivized experiments, Dietvorst, Simmons and Massey found that people were considerably more likely to use an imperfect algorithm, and performed better, when they could modify its forecasts even slightly.

Kolassa also notes that end users typically prefer an additive breakdown of what moved a forecast. A statistical baseline with small adjustments on top lends itself to exactly that.

Where this could be wrong

Machine learning won M5. Cerqueira, Torgo and Soares found that it improves relative to statistical methods as sample size grows, and argue that the statistical advantage holds only at extremely low sample sizes. The authors of the 2018 study also warn that their results may differ when nonlinear patterns or other factors dominate a series.

Simple methods also have limits the design has to cover. Kolassa points out that Walmart relies very little on promotions, which makes its sales easier for exponential smoothing, while most retailers do rely on them. The weather gains come from one retailer and were largest in specific conditions. Both results suggest using expert models selectively, but neither tests our design.

What this does and does not show

This evidence supports starting from a statistical baseline and adding complexity only where it earns its cost. It does not prove that 80% is the right split, and none of these studies measures Seamore. We have no accuracy results of our own yet. If our tests disagree with the split, the split should change.

Read how the engine is designed and what is still open →

If you are working on the same problems, or have operational data these methods should be tested on, get in touch.