Statistical and Machine Learning forecasting methodsConcerns and ways forward

Let's start with the promise. For years, machine learning in forecasting, or ML, has come with a confident headline: smarter algorithms should beat old-school statistics. Then the tests got tougher. As Makridakis, Spiliotis, and Assimakopoulos put it, there was plenty of advocacy and not much objective evidence, especially once you look beyond the next month or two. So, they designed a clean, competition-style trial: same data, same horizons, and apples-to-apples comparisons of accuracy and compute. The stage was the M3 world, but focused. Instead of thousands of mixed-frequency series, they worked with one thousand and forty-five monthly time series that Ahmed and colleagues had analyzed before. These were reasonably long for monthly data—eighty-one to one hundred twenty-six months—enough history to train, then eighteen months set aside for testing. The training window was always the first n minus eighteen observations; the target was the next eighteen, predicted all the way out. And because neural networks can twitch depending on initial weights, they repeated the forecasts ten times and averaged the errors so a lucky start didn't masquerade as a great model. To score performance, they leaned on two workhorse metrics. The symmetric mean absolute percentage error, or sMAPE, averages the absolute percent gap between the forecast and actual but scales it so errors up and down are treated evenly. The mean absolute scaled error, or MASE, divides your absolute errors by the average one-step error of a naive benchmark, which makes the numbers comparable across wildly different series. They also tracked two pragmatic quantities that forecasters care about but papers often skip: computational complexity, defined as time relative to a seasonal naive method, and a model-fit score that summarizes how tightly a model clings to the training data. The point of that last one was to probe a classic worry: if a model hugs the past too hard, does it stumble in the future? Then they threw a wide net across methods. On the machine learning side, they explored multilayer perceptrons, or MLPs, Bayesian neural nets, radial basis and generalized regression networks, k-nearest neighbors, regression trees, support vector regression, and Gaussian processes. They also pulled in two of the architectures that dominate modern deep learning—recurrent neural networks and long short-term memory networks, or LSTMs—to ask the obvious question: do recurrent structures that shine in language or audio help in these monthly business and economic series? On the statistical side, the classics showed up: seasonal naive, often called Naive 2, simple exponential smoothing, Holt and its damped variant, exponential smoothing with state-space estimation, combinations of forecasts, AutoRegressive Integrated Moving Average, or ARIMA, and the Theta method that famously did well in M3. Preprocessing wasn't an afterthought; it was a controlled part of the experiment. Ahmed and colleagues had explored a grid of transformations and found a pattern that will sound familiar if you've had to forecast anything seasonal. Stabilizing the variance with a Box–Cox transform was a hair better than a straight log in terms of sMAPE, but neither moved mountains. The big lever was seasonality. When they deseasonalized, the one-step sMAPE dropped from about nine point fifteen to eight point sixteen, and MASE from zero point sixty-seven to zero point fifty-seven. Stack Box–Cox on top of that and you got the best sMAPE combination; add detrending to the mix and you nudged MASE a bit further down. Preprocessing didn't turn a weak learner into a star, but it made the playing field flatter and the results more stable. So what happened when all of that met the scoreboard? The headline is blunt. Across one-step and multi-step horizons, the best statistical methods outperformed the machine learning entrants and did so while using far less compute. And this isn't one stray metric talking. In the one-step comparisons—what many practitioners care about—Ahmed's earlier paper saw the top neural net, a multilayer perceptron, at about eight point thirty-four percent sMAPE, with Bayesian neural nets and Gaussian processes close behind and methods like radial basis networks and trees trailing into double digits. In the extended tests, with careful preprocessing, the best neural model posted an sMAPE of eight point seventeen percent, and the multilayer perceptron came in around eight point thirty-nine. Respectable. But ARIMA, a workhorse with half a century of history, clocked in at roughly seven point nineteen. That gap—about one percentage point—doesn't sound dramatic until you remember it's averaged over one thousand series. Even more telling, in some setups the best machine learner beat the seasonal naive baseline by only zero point nineteen percentage points. If a seasonal random walk is nipping at your heels, it's hard to argue for a heavyweight neural net in production. As you push beyond next month, the pattern holds. ARIMA often led at short horizons—think one to six months—with sMAPE around eight point ninety-three and MASE near zero point sixty-one, while Theta and simple combinations of statistical forecasts rose to the top for medium and longer ranges. The machine learning strategies designed for multi-step forecasting—the iterative approach that feeds each forecast back in as input, the single network that tries to spit out all eighteen months at once, and the bank of eighteen separate horizon-specific nets—didn't change the story. Sometimes the direct or multi-network schemes were even worse than the humble iterative loop. Recurrent models, which you might expect to thrive on sequences, didn't deliver systematic gains in this setting. Let's pause on compute, because it's where the rubber meets the road. The team normalized run times to the seasonal naive and asked a practical question: how many multiples slower is this approach? Traditional statistical methods landed in the single digits or low tens. Some neural configurations, especially the "train eighteen nets for eighteen horizons" approach, shot past four hundred times slower. Yes, you can prune training iterations or trim hyperparameter hunts, and they did, but the gap stayed lopsided. When your organization needs to run thousands of forecasts nightly, that difference isn't an academic footnote; it's a budget line. Why did the neural nets lag? Part of it is the data regime. These are monthly series with a handful of years of history, not minute-by-minute traces with millions of points. Nonstationarity, changing seasonality, and noise make it easy to memorize quirks that don't generalize. The authors showed that in-sample fit and out-of-sample accuracy were often mismatched. A multilayer perceptron that hugged the training data with a model-fit error around two point eleven percent still missed future values more often than ARIMA, which fit the past a bit worse—about two point fifty-nine percent—but forecast the future better with a seven point nineteen percent sMAPE. That disconnect screams for careful stopping rules and regularization. It also reminds us that in classic time-series land, parsimony—the idea that a smaller, well-specified model can outperform a big flexible one—isn't a slogan, it's a survival skill. There's also a historical backdrop to all this. The M3 Competition, led by Crone, Hibon, and Nikolopoulos, was the first large-scale, many-series testbed where neural nets and statistical methods went head-to-head. The Theta method topped the charts, and while a handful of nets beat an older adaptive architecture on some counts, there was no sweeping machine learning victory. Ahmed and colleagues picked up that thread, added recurrent models like LSTMs, compared eighteen-step forecasts, and layered in compute and model-fit diagnostics. Their synthesis was consistent: a few neural flavors—multilayer and Bayesian—hung near the front of the machine learning pack, but the field as a whole didn't surpass the strongest statistical baselines. And, critically, results were sensitive to choices that sometimes get glossed over: which horizons you score, which error metrics you report, and how you preprocess. That sensitivity is a warning and a help. It's a warning because if you cherry-pick a horizon or a metric, you can convince yourself something's better when it's not. It's a help because it points to what matters operationally. Deseasonalize your series and you'll give any method a cleaner shot at the pattern. Use MASE alongside sMAPE so you're not misled by scale. And benchmark, benchmark, benchmark. As Makridakis and colleagues argued, competition-style evaluations with common baselines and open replication are how we cut through anecdote. The computational story suggests deployment trade-offs, too. If a damped Holt or a combination of a few statistical forecasts is both accurate and fast, they make a compelling default, especially when complexity doesn't buy you much. In some of the experiments, a carefully preprocessed ensemble of simple statistical methods delivered the best balance: low error numbers with runtimes you could laugh off. Meanwhile, the multi-horizon neural strategies that feel so modern—direct multi-output networks, banks of horizon-specific models—didn't confer the hoped-for edge. More outputs and more parameters didn't translate to better extrapolation. It's also fair to ask about uncertainty, because a point forecast without a sense of spread is only half a forecast. The authors noted that many machine learning entries were black boxes without reliable interval estimates. That's not a dealbreaker by itself, but it adds friction if you're trying to plan inventory or staff hospitals. Classical methods often come with uncertainty machinery baked in—state-space exponential smoothing and ARIMA being prime examples—so even when point errors are similar, the decision value can tilt toward the tool that gives you honest intervals. Now, caveats. These results are about a particular slice of the forecasting world: monthly series, modest lengths, and plenty of seasonality. The broader M3 set spans frequencies and domains, and the next competition, M4, scaled up to one hundred thousand series to shake out where methods win and why. The authors themselves stress that different data or longer histories could tilt the field, and they're not claiming machine learning can't win anywhere; they're saying it didn't win here, under rigorous testing, and when it did well, the gains were small and costly. So where does this leave you if you have to forecast something real? Start simple and earn complexity. Automate preprocessing—Box–Cox when variance wanders, deseasonalize when the calendar's in the driver's seat. Put a strong statistical baseline on the board—Theta, an ARIMA that's carefully selected, or a small combination. Then, if you try a neural net, evaluate it exactly the same way across the same horizons and metrics, and watch for overfitting like a hawk. Treat compute as part of accuracy; a model that's one point better but four hundred times slower isn't really better unless you have a special reason. The broader message is less sexy but more durable. Forecasting remains a domain where structure and parsimony, plus transparent evaluation, beat raw function approximation. As Ahmed and colleagues, and before them the M3 team, showed, the gains come from design discipline: preprocess what you know, measure what matters, and don't fool yourself with in-sample fit. And if you want to push machine learning forward here, build on that foundation—regularize, stop early, quantify uncertainty, and test on bigger, broader datasets. That's why the community leaned into competitions. M4, the successor to M3, expanded the canvas dramatically—more frequencies, longer series, one hundred thousand real-life cases—precisely to generate the kind of objective evidence this debate needs. It's the right impulse. Because once you put all the methods on the same track, under the same clock, what works tends to reveal itself. And sometimes, as this story shows, what works best is the thing that's been quietly working for decades.