Fundamental Limits of Foundation Forecasting Models: Multimodality and Rigorous Evaluation

This post is based on Christoph Bergmeir Yes. NeurIPS 2024 The speech. "Fundamental limitations of foundational forecasting models - The need for multimodality and rigorous evaluation"。

Professor Christoph Bergmeir is one of the main maintainers of Monash Time Series Forecasting Library of Monash Series Forecasting Repository.

The questions in this article can also be addressedLinear time series analysis: smooth sequence, ARMA and ARIMA、Financial time series analysis: ARCCH/GARCH effect and volatility modellingHow the concept of a relatively close read together is developed in different contexts.

Introduction: Start with random travel

Time-series prediction is one of the most fundamental but also the most difficult topics in the data sciences. Unlike image or natural language processing, time series often face extremely low levels of belief.

If it's a completely random sequence of Random Walker -- there's no hidden pattern -- - So the best prediction of the sequence is... Naive Forecast, that is, the first-minute observations are projected at the next-minute projection ($y_{t+1} = y_t$I'm not sure. In such cases, however complex models (e.g. BP Neural Networks, SVM, Random Forests, etc.) are often less performing than the simplest of the Naive methods.

This seemingly simple logic is often overlooked in today ' s in-depth learning prediction studies.

Financial “false SOTA”

Stock markets are a typical example of random migration. The EMH (EmH) is of the view that stock prices are not a function of past prices, but a reflection of future expectations. The information contained in the equity price is almost already reflected in the current public information, and future changes are mainly influenced by new and unpredictable information.

Thus, stock prices are often considered to be of a moistique (Martingale) nature. The blogger says that the government is not a party to the law.The accuracy of the Naive prediction is almost impossible in theory.I'm sorry. In fact, quantitative research in the financial field tends to focus less on mere predictions (i.e., what the stock price is tomorrow) than on more.Risk (Risk) andVolatility。

Many articles published at the top-of-the-art conference claim to have achieved SOTA (State of the Art) in the area of financial forecasting. But looking at these papers, they tend to be compared only with other in-depth learning (DL) methods, while ignoring the strong benchmark of the Naive prediction. The calculation time of the model is expanding, but performance does not necessarily lead to a substantial breakthrough.

Weather predictions and wrong benchmarks

In addition to financial data, in-depth learning researchers are also keen to forecast weather and electricity data. But there is also a common sense error.

The meteorologists generally believe that due to the confusion effects,Long-term weather predictions over two weeks (14 days) are physically impossible.I'm sorry. Thus, any model that claims to be capable of producing a long-term (greater than two weeks) accurate hour-by-hour weather prediction is largely acoustic or random speculation.

The paper found that they did compare traditional statistical methods such as ARIMA or EDS, but that benchmarks were often inappropriately set. For example, when faced with weather sequences with complex seasonal (e.g. hourly data), simple ARIMA is not the right opponent. The stronger benchmark should be DHR-ARIMA (Dynamic Harmonic Regression with ARIMA errors)I'm sorry. The introduction of this statistical model, which is more complex but more suitable for this type of data, often leads to the failure of the so-called SOTA deep learning model.

Assessing the trap: Drop Last Trick

In order to make their models look better, some scholars have even been brain-drived in the assessment process. A typical example is... "Drop Last Trick"。

In many of the time series of in-depth learning, data are processed in multiple Batchs. If the last Batch of the test set is not satisfied, some of the coding libraries (such as those that were wrongly configured) DataLoaderIt's a big deal.

However, in time series projections, the data are orderly.The last part of the test set is often the most up-to-date, closest and most relevant dataI'm sorry. The arbitrary discarding of this part of the data could lead to a serious distortion of the results of the assessment. In many articles, the same method has a wide variation in performance in different papers, often because different (even less stringent) test criteria are used to create so-called SOTA, and the actual effect may be much less than in Baseline, decades ago.

Double-edged global Models Sword.

In recent years, a large number of multi-source time series have been used to constructGlobal Models It's becoming a trend. Only in-depth learning techniques can effectively process this mass of data.

Studies have shown that even training in global models on irrelevant data, with further fine-tuning of the field, may be better than local models. This idea is a long-standing counterpart in basic statistics, called James-Stein ParadoxI'm sorry. It allows us to improve the predictions using irrelevant data, and to reduce them by introducing deviations (Bias) in exchange for differentials (Variance), which is the basis of the theory of normality.

However, models that are valid across the board do not guarantee their effectiveness in specific local data.

The current large time series model (Foundation Mode for Time Series) often faces a problem: Algorithms combine data from various sources during learning. This “averageization” approach has resulted in models not being able to predict accurately in combination with real scenarios and corresponding training data.Algorithms can average different hidden models instead of using them separately.; data sources are also being erased.

While language models are also taught through pre-training of common language models, we can correct models through Prompt in a round of dialogue. This “immediate correction” is much more difficult in the pure time series projection.

Way out: Context is King

So, what's the next step in the base prediction model?

The blogger says:Simple time series lacks enough information to be usedI'm sorry. If we rely on historical data alone, it is difficult to break the random travel restrictions.

♪ Want to be in LLM4TS (Large Language Models for Time Series) It's a problem that needs to be addressed. ContextI'm sorry. The context is not just a longer historical window, but a longer one.Multimodity The introduction of information - including text news, macroeconomic reports, image data, etc.

Only when the model understands the text of the “corporate scandal” Context will it be possible to predict a collapse in stock prices; only when the model combines real-time weather cloud maps can it break the bottleneck of pure numerical prediction.

This is the more feasible route for the TS baseline model: from the numerical combination of a single modulus to the context of a multi-modular modulation.