How little machine learning does a good forecast need?
We are building a forecasting engine that is about 80% statistical models, with a thin layer of small experts and routers. It is still in development. This is the reasoning behind it, and what we have not worked out yet.
/Forecasting/3 min read
What the competitions show
The M competitions are the closest thing forecasting has to a public test. M4 compared 61 methods on 100,000 time series. Twelve of its 17 most accurate methods were combinations of mostly statistical approaches, the winner was a hybrid of statistical methods and machine learning, and none of the six pure machine learning methods beat a simple combination benchmark.
The 17 most accurate methods
12 were combinations of mostly statistical approaches
The six pure machine learning methods
None beat the simple combination benchmark
M5 points the other way. Its 42,840 series are Walmart unit sales, organized in a hierarchy down to item and store, and its top entries were machine learning models, led by LightGBM ensembles trained across many related series.
We do not read this as one side winning. We read it as the answer depending on the data: how much history a series has, how many related series sit beside it, and what outside information is available.
What we are building
About 80% of the engine is statistical models: exponential smoothing, ARIMA, Theta and forecast combinations. They are cheap to run across a large catalog, they work on short histories, and a planner can inspect what they did. The 80% split is a design choice, not a measured result.
Read the evidence behind this choice →
The thin layer on top is small experts, compact models that each handle one kind of real-world signal, such as weather, price and promotion changes, shipping disruptions or competitor moves. Routers decide which experts apply to a given forecast. The idea of letting specialized models take different parts of a problem goes back at least to mixtures of local experts, published in 1991.
Company data
Mapped and cleaned
Statistical models
About 80% of the engine: exponential smoothing, ARIMA, Theta and forecast combinations
Real-world signals
Weather, prices, shipping, competitors
Routers and small experts
A thin layer: routers choose which experts apply, and each expert handles one kind of signal
Forecast
What is still open
Routing is the hard part: which expert to trust for a given product, horizon and demand pattern, and when to leave the statistical forecast alone. An expert that fires too often adds noise to a forecast that was already fine.
Uncertainty is the other. M4 scored prediction intervals as well as point forecasts, because a planner setting safety stock needs a range they can rely on, not just a number. How to keep that range honest once experts adjust the forecast is unsolved for us.
Where this stands
The engine is not finished, and the forecasting screens in the application show illustrative projections for now. We have no accuracy results to publish yet. When we do, each will state its data, horizon and baseline.
If you are working on the same problems, or have operational data these methods should be tested on, get in touch.
References
- Makridakis, Spiliotis, Assimakopoulos (2018). The M4 Competition: Results, findings, conclusion and way forward. International Journal of Forecasting 34(4), 802–808. DOI
- Makridakis, Spiliotis, Assimakopoulos (2020). The M4 Competition: 100,000 time series and 61 forecasting methods. International Journal of Forecasting 36(1), 54–74. DOI
- Makridakis, Spiliotis, Assimakopoulos (2022). M5 accuracy competition: Results, findings, and conclusions. International Journal of Forecasting 38(4), 1346–1364. DOI
- Jacobs, Jordan, Nowlan, Hinton (1991). Adaptive Mixtures of Local Experts. Neural Computation 3(1), 79–87. DOI