Nixtla's Open Source Time Series Ecosystem.

Nixtla Community

hey <@U052AN4HV24>! Thanks for using neuralforecast, the problem probably arises due to the newest release of torch (2.0.0), could you try installing “torch&lt;2.0.0”? We are working on supporting the newest version

Hey <@U050ZLJCYMD>!

We have a tutorial on how to use neuralforecast using Walmart’s sales: <https://nixtla.github.io/neuralforecast/examples/intermittentdata.html>

And here you can find an example of how to compare and ensemble statistical and neural methods using the same data: <https://nixtla.github.io/statsforecast/examples/statisticalneuralmethods.html>

happy forecasting :)

Hi <@U0517FJ5K54>, we are currently working on creating this function. In the meantime you can use the `predict_rolled` function. Another user had the same question and we discussed how to do it an a thread, I will tag you.

Most point prediction losses can work:
MAE, MSE, sMAPE

And other distribution prediction losses:
MQL, DistributionLoss-Gaussian, Gaussian Mixture

You can feed your features into the model through hist_exog, futr_exog and stat_exog lists

Yes but the model doesn't improve on Validation loss

Not all features need to have predictive power
Some can also have redundant information that makes learning hard

NeuralForecast does not have automated feature selection capabilities yet

hey <@U04GUDQLHR8>! Thanks for using neuralforecast. We have experienced that behavior due to the huge amount of logs output by ray, is that the case? The workaround is simply to put %%capture at the beginning of the cell that runs `fit`  or `cross_validation`

Hey, Thanks I will try that, it does indeed sounds like that could be the reason.

Cool! Let me know what happens, we are still trying to figure out how to get rid of those prints xD

Thanks, I will do that. It is running now :slightly_smiling_face:

There is also a lot of outputs, so I can why you wanna do that or at least control it someway.

<@U0316L4HWQ7> Thanks for the suggestion, it seems to work.

hey <@U052AN4HV24>! Yesterday we updated the library to make it compatible with torch and lightning &gt;= 2.0, could you try installing the new version from the `main` branch? Here’s the line: `pip install "git+<https://github.com/nixtla/neuralforecast.git@main>"`

I tried to install the new version with pipenv as i am working with Pipfile, getting an error like
`Exception: Could not deserialize ATN with version  (expected 4).`
when importing neuralforecast in Python

hey <@U050ZLJCYMD>! Do you know if the error arises only with the new version or also with the previous ones? I’m thinking that maybe it is related to the python version; what version are you using?

<@U0316L4HWQ7> This error arises only with the new version. Running locally on Mac M1
The previous `neuralforecast` with `torch==2.0.0` throws the same error which was stated by <@U052AN4HV24>

Hi team, quick question related to this: is there an estimated release date into PyPI for the next NF version with this fix?

hey <@U039UR72YLE>! We are still making some improvements for the next release. I think they should be ready by the end of the next week. WDYT <@U03619XRWKD>?

<@U039UR72YLE>, Jorge told us that you had a problem, is it fixed now? Do you know how we can help with that? :slightly_smiling_face:

Hi <@U04UKKD2F4K>! Yes, all variables are padded at the beginning. We have not tried the "real_data" dummy, but it sounds intuitive. We previously had an "available_mask" dummy to directly mask missing/padded data with zeros. This is a little bit different but related. The scaling is done after padding the variable, but it shouldn't affect the information provided by the variable

Also note that most models use only the information in the input window of size `input_size`. If the time series have a lot of history then it should not be necessary to add the "real_data" variable.

<@U03619XRWKD> Thank you! Yes the problem in my specific case is that many time series have a limited history which is shorter than the input_size I need for modeling the yearly seasonality in an acceptable way.

Background provided by <@U03DDFKMT0X>:

Background:
• we have a feature store of ~5M features from power markets
• some of those are forecast features, others are actuals
• we can target any actuals series, and combine with future regressors (forecast features) and lagged regressors (actuals features)
Findings:
• when we generate datasets with mostly lagged features, TCN vs NHITS accuracy wise isn't that noticeable
• when we have lots of future regressors in the dataset, TCN gets a noticeable accuracy pickup, still need to quantify how much but early estimates are ~15-20% MSE improvement, will calculate RMSSE shortly for target-wide evaluation
• TCN trains considerably faster for the same dataset and target

"biggest takeaway for us was NHITS is probably best in class if you have no future regressors, but if you do, you may want to try TCN :slightly_smiling_face:"

"we think TFT may offer the same benefits but tbh it's so slow to train we haven't been able to test it"

<@U038ZT9UDGA>, this is something you might be interested in commenting.

Good to confirm that future alignment works
<@U03DDFKMT0X>, <@U03619XRWKD> 

We need to include the future alignment into the NHITS

Additionally the original TCN had different types of contexts that resemble the interpolation principles of NHITS that we could also explore

And regarding TFT speed 
It is due it not including the forking sequiences optimization technique

As an aside, while we were going through all of this we stumbled upon <https://github.com/ogrnz/feval> - super helpful to sort out which set of forecasters perform the best

Yep
We already had GW, DM and MCS tests in NeuralForecast 0.x

It got lost in the transition to NeuralForecast 1.x

I will check the package, thanks for the pointer

Thanks for sharing the results <@U03DDFKMT0X>! Indeed, the NHITS currently only concatenates the exogenous variables as inputs of the MLPs. If you have many features then MLPs might struggle to learn all the effects (maybe you will need to increase its size considerably). Adding the decoder as the TCN that receives future values can definitely help.

I am still unsure of what is the best general approach. Pure "future alignment" might fail to model effects between observations of the horizon window (eg. a large spike in the future might decrease values now). In our NBEATSx paper adding the exogenous variables alone as inputs of the encoder achieved amazing results. Probably a combination of both approaches will work best in general, with the global context vector, or attention mechanisms such as TFT (but it adds a lot of computation time)

Interesting. But in practical terms, it's rare to have 5M features in a time series problem.

For the sake of clarity - we have *access* to ~5M features, a mix of lagged and future regressors, which was what allowed us to stumble onto this finding. We definitely don’t use all features in a single model :blush: 

I see. That makes more sense :sweat_smile:.

Hi <@U05378ZMZ96>. In our experience TFT is highly memory intensive. You can limit the memory usage with `windows_batch_size` and `hidden_size`  parameters

You can also try to reduce the size of the autorregresive features through the `input_size`

Hey <@U05378ZMZ96>,

We fixed this week TFT's memory usage: <https://github.com/Nixtla/neuralforecast/pull/597>
We will merge that PR into main soon.

If you want to try the changes, you can install them with this line:
```!pip install git+<https://github.com/Nixtla/neuralforecast.git>```


image.png

Hi <@U052NCY2PM0>,

With the update towards PL 2.0, the validation_epoch_end was deprecated.

If you pip install from the main github branch this issue should be solved.
!pip install <git+https://github.com/Nixtla/neuralforecast.git|git+https://github.com/Nixtla/neuralforecast.git>

Hey <@U038ZT9UDGA>, thanks for the quick reply.
I have tried installing dependencies from the updated git location as well, still facing the same error:

Screenshot 2023-04-11 at 8.46.19 AM.png

Sometimes Google Colab needs you to update the installation by pressing this button:

It has a very weird memory of the python environment

Yes, it worked after disconnecting the runtime.. thank you!

Hey <@U0517FJ5K54>,

I was working a month ago on the DeepAR model inclusion, and left the project unconcluded.
For the moment the only thing that is missing to have a DeepAR is to work on a recurrent decoder that uses LSTM rather than MLPs.

For the moment  we have an `LSTM` model that operates closely to the DeepAR with the decoder difference.
Here is its documentation:
<https://nixtla.github.io/neuralforecast/models.lstm.html>

Out of curiosity are you working on baselines for a paper or are you trying to get good performing methods?

It is my belief that DeepAR prediction's accuracy and speed have been surpassed by various methods already available in NeuralForecast.

<@U038ZT9UDGA> Thank you for your response! Yes, I am working on reproducing the results of a paper. So, if

You may want to check this GluonTS example:
<https://ts.gluon.ai/stable/>

If i replace the mlp_decoder with a lstm_decoder in lstm.py, then I would get the DeepAR model? Moreover, which other methods would you suggest?

If you aim to achieve recognition in the quality of your baselines, I would go for NBEATS (Bengio), TFT (Google).

If you want to help us with citations, and use good performing models NBEATSx, NHITS are good baselines too.

LSTM is almost needed, as is one of the most classic methods

And if you want to increase the seriousness of the baselines, I would include an ARIMA/ETS

Got it! Thanks so much! Do you think it is straightforward to replace the mlp_decoder with a lstm_decoder in lstm.py?

There are still several differences to how DeepAR works, so it wont be completely comparable. DeepAR simulate trajectories with a recursive strategy, our method produce all the forecasts simutanously. Also I believe DeepAR does not have a decoder, it is basically a simple LSTM.

It is a Seq-to-Seq method in its purest form

Yes, but I what I meant <@U038ZT9UDGA> <@U0517FJ5K54> is that you dont need to "add" an LSTM as decoder instead of MLPs, because DeepAR consists of only one LSTM. You will need to remove the MLP decoders and only keep de LSTM, and use a Distribution Loss. Still, because of the recursive strategy and sampling it wont be exactly the same.

Hi <@U0539TUMNSG>. It is not possible right now to only use information from the covariates. This is because during training the loss is computed with the values on the target variable `y` , but there is no way of disabling its information in the inputs.

You could modify the `forward` step of the models and remove the historic values of `y` (when they are concatenated with the historic covariates), or replace its values to 0.

Is there a reason why you may not want or have access to the autorregressive features?

A suggestion in case you don't have access to the autorregressive feature is to run a simple regression model against the covariates.

Note that not having autorregressives could potentially undermine the effectiveness of most forecasting methods.

Thanks for the suggestions. my use case was feeding the TCN model with decomposed components of my target (using VMD) as covariates, rather than the target itself. I'll have a play to see if I can get it working.

20230417_132819.jpg

Hey <@U04UKKD2F4K>,

How are you evaluating the predictions? Or is this a qualitative evaluation?

On the side of controlling the seasonality using a multiplicative approach.
It should be possible to create a Box-Cox log scaler in the <https://nixtla.github.io/neuralforecast/common.scalers.html|TemporalNorm> module.
Using a logarithmic transformation (multiplicative deleveling) could reduce the predictions variance.

Another idea that has given me good results is to add a `SeasonalNaive`  feature to better control the seasonality in the  `futr_exog_list`  parameter.

Let me know your thoughts.

<@U038ZT9UDGA> Thanks! Previously I had tried to log transform the data, perform the forecast on the transformed data and then reverse the log transform with exp, but it did not seem to work well (I was getting overly smoothed forecasts). About the last idea, would you use a naive seasonal model to perform a forecast and then use that forecast as an exogenous variable for NHITS?

Yes, use Seasonal Naive as a model input. To help it better model the seasonalities.
Its almost like modeling the I residuals vs SeasonalNaive, in an AR I MA model.

Hi <@U0517FJ5K54>! Are you using the defaults for the other hyperparameters? What input_size are you using?

Thank you for the response! I am using nbeats_config = {
        “input_size”: tune.choice([2*seasonality]),
        “learning_rate”: tune.loguniform(1e-4, 1e-1),
        “scaler_type”: tune.choice([None, “robust”, “standard”]),
        “max_steps”: tune.choice([500, 1000]),
        “batch_size”: tune.choice([32, 64, 128, 256]),
        “windows_batch_size”: tune.choice([128, 256, 512, 1024]),
        “random_seed”: tune.randint(1, 20),
        “val_check_steps”: tune.choice([1]),
        “early_stop_patience_steps”: tune.choice([10]),
        “max_epochs”: tune.choice([100])
    }

models=[
    AutoNBEATS(h=horizon, config = nbeats_config, loss = MAE(), refit_with_val=True, num_samples=10)
        ]

and the Wiki2 dataset from the hierarchicalforecast package.

From what I understood:
• You have stark scale differences between bottom and top levels.
• Due to the scale difference, absolute metrics like RMSE/MAE prioritize the top levels.
• You used SMAPE to help with the top-bottom scale difference.
• You wonder if there are weighted losses in NF
Answering the questions:
• Like SMAPE, probabilistic losses (DistributionLoss, PMM, GMM...) are scale independent by definition.
• Regarding weighted losses, check scale-independent errors and weighted MQLoss here is the <https://nixtla.github.io/neuralforecast/losses.pytorch.html|NeuralForecast losses catalog>.
• Before ditching the probabilistic approach I recommend you to carefully explore:
    ◦ Use <https://nixtla.github.io/hierarchicalforecast/examples/tourismsmall.html|HierarchicalForecast tutorial> to set an accuracy expectation.
    ◦ Explore your forecasting features, control for seasonalities if any.
    ◦ Model hyperparameters (learning rate and random seed in particular).
Hope this helps.

<@U038ZT9UDGA> Thanks! I have yearly seasonality, what do you mean by "control for seasonalities"?

You can add month dummies, like the ones in this example:
<https://nixtla.github.io/neuralforecast/examples/hierarchicalnetworks.html>

Hi <@U04DB0W0T0D>. Yes! We have a tutorial on Transfer Learning here: <https://nixtla.github.io/neuralforecast/examples/transfer_learning.html>. You can pretrain a model and then use it on any other dataset without further changes.

hey <@U053D6K21LZ>! Thank you for letting us know about the issue. Do you happen to have more information about the environment you’re using? (python version, torch/lightning/neuralforecast versions would be great) :)

Thank you <@U0316L4HWQ7> we  are running on python 3.10, torch 2.0.0 , ligthning 2.0.0 and neuralforecast 1.5.0. OS is Linux and we are using azure function (Microsoft serverless compute service). Thank you again for your kind help


A quick update, i have tested all the neural method (AutoMLP,NBeats,NHiTS…) and I am always facing the same error ( try to log-1…)
I also downgraded lightning to 1.9.5. Same problem. <@U0316L4HWQ7>  do you have any ideas ? Thank you very much 

hey <@U053D6K21LZ>! Were you able to solve the problem?

Hello <@U0316L4HWQ7>  unfortunately not. I have tried multiple combination of torch and lightning version…I think the issue is more related to the architecture we are using (azure function) and the ability to have write access and log file. We will continue investigating. I have not seen other people using azure function with neural forecast. I will keep you posted if we find a solution 

Hi <@U055L8LSF6V>! In time series forecasting it is crucial to avoid information leakage, that is, when doing a forecast we can only use the available information at that time. We have the historic/future variables division to allow for different types of covariates, depending if we know their future values or not.

For instance, if today (04/27) you are doing the daily fuel sales forecast for tomorrow (04/28), and you know the prices for tomorrow, then they are future exogenous variables. The models can use the tomorrow's price to produce the forecast. If you only have past values of the prices (up to 4/27), then they should be historic variables

Hey <@U055L8LSF6V>,

Thanks for checking out HINT, in the <https://arxiv.org/pdf/2110.13179.pdf|DPMN paper> (main HINT inspiration) what I did was to carefully craft aggregate level exogenous.

For example I used static dummies from the location hierarchy to capture geographic effects.
It is possible to give that information to HINT through the `static_exog_list`parameter and the `static_df`  inputs.
In your specific example, you can transform rural/urban dummy into an average of urban population as an aggregate measure.

Hope these ideas help.

Complementing Kin's answer, note that you still need to define values for all exogenous variables for all time series. The base model of the `HINT` class operates as any other `nf` model. If there is no obvious aggregation rule for the exogenous variables, it might also work to simply set missing/not defined variables to 0 (or other value which does not collide with true values of the lower level variables), which is equivalent to "masking out" the missing data.

Hey <@U04VD4PFER2>,

Would you be able to provide more insights on your work and the bugs you get.
If you put the error in a github issue along with a jupyter notebook that replicates the error it can be easier to help.

Having a variable sampling rate is an interesting issue. My approach in the past has been resampling the signal at a fixed rate. But I would be very interested to know if there is a forecasting technique that can natively handle signals with time-varying frequencies.

Hi <@U055D7UT6TC>! The current methods do not natively incorporate the time delta between timestamps to make predictions. They fundamentally assume the time delta between samples is constant. Having unevenly spaced time series will lead to unexpected behaviour, probably producing inaccurate forecasts. I suggest, as Farzad mentioned, to resample the signal.

<@U040AMG6N69>, could you point me out to examples on how to resampling the signal at a fixed rate?

I don't have an example handy. I used to do it in MATLAB years ago when I was doing signal processing. You need to either upsample or downsample to get to a uniform frequency. I haven't done it in Python but I'm sure there are libraries for it. If you were going to write your own code you would take the largest time step you have and interpolate your signal at those intervals for downsampling. Numpy does the interpolation if you give it your timestamps. For upsampling you can take the smallest time interval you have and repeat the same process. You can also choose any time interval between your smallest and largest.

Thank you for your advice <@U040AMG6N69>!

Hi <@U055L8LSF6V>. Yes, all methods can receive time series from different length. Within a training batch, shorter time series are padded with 0s at the beginning, to match the length of the longest time series.

Hey <@U055QNWRLRG>,

Thanks for using NeuralForecast :slightly_smiling_face:
This is currently among one of our priority fixes in NeuralForecast inference.

A partial fix was already implemented allowing you to use the `inference_input_size` , parameter that effectively trims the length of the time series to avoid using them entirely. I would recommend setting it to `inference_input_size=test_length*3` .

Let me know if this helps.
Would you be able to report this in a github issue if your problem persists?

Good to know <@U055QNWRLRG>

Would you be able to post this as an issue on github?
That way if somebody has similar issues they find the solution :)

It looks like for fitting RNN, memory consumption allocated is still orders of magnitude larger than what is implied by the number of parameters. For example, using a dataset of length 10000 with 14 endogenous time series using default network &amp; batch sizes uses much more than my 12GB capacity. Have you tried running nf.fit() on dataframes of similar size, and if so, what was the peak memory usage?

Hey <@U055QNWRLRG>,

We might have a curse of dimensionality problem showing up.

Our RNN-based methods use a forking sequences optimization technique that creates the windows of size [series, time, channels, horizon].

Possible ways to alleviate the memory usage:
• reduce the series batch size.
• filter the length of the series of your data directly ir with input_size and inference_input_size
• change to WindowsBased models like MLP, NBEATS, NHITS
From our side we need to batch at the time level the creation of the forecasts.

Thanks for your info, I will take a closer look at the source code to get a better idea of how this technique works in practice!

Most models have early stopping by default, you can control its definition through ‘early_stop_patience_steps’, ‘valid_loss’, ‘valid_check_steps’.  Here is the documentation of the NHITS model.
<https://nixtla.github.io/neuralforecast/models.nhits.html#nhits>

Additionally if you are interested in the source code check this example of the inner workings.
<https://github.com/Nixtla/neuralforecast/blob/main/neuralforecast/common/_base_windows.py#L126>

Hi <@U055TNLAABG>! Thanks for using our library! Regarding your questions:
1. The models do not learn unique_ids, nor use them to forecast. Models can learn time-series specific dynamics only if you add static variables (such as one-hot encoding of the unique_id).
2. During training, each batch is created by randomly sampling a subset of `batch_size` time series, and `windows_batch_size` windows from the subset. We dont have a parameter to limit the total number of windows from each time series. However, we dont usually train models with all possible windows from each time series, we recommend to train models with a fix number of training steps (`max_steps`). 

Regarding 2: My full dataset comprises of multiple small datasets and each small dataset has multiple time series. So rather than combining the small datasets I should train the model one by one separately on each small dataset using max_steps so that the model does not learn too much from one dataset. Does this approach make sense to you ?

This is a very interesting question. I believe it would be better to pool all the data together, so that the model does not end specializing more to forecast the latest datasets. Usually is better to have variety in each batch during the training step, than grouping potentially similar time series in each batch. In competitions such as the M4, high performing models such as the ESRNN and NBEATS were simultaneously trained on many time series from different domains and sizes.

In the near future we will also add the possibility to add weights to each unique_id and timestamp for gradient computation

I was hoping to do that as well but there are some datasets which have significantly longer and more number of time series than others so more number of samples which would lead the overall dataset to become imbalanced. I could potentially randomly sample some fixed number of time series from larger datasets and then combine the smaller datasets to form a larger dataset. Would this be an ideal approach or should I think of something else ? Your thoughts are always appreciated

Screenshot 2023-05-04 at 10.30.17 AM.png

Hey <@U04UKKD2F4K>,

Thanks for using NeuralForecast.
*Can you open a github issue to have this in my TODO list?*

In the past we had `valid_loss=loss`, as a default for which we did not include the possibility to include `loss`, as a tunable hyperparameter. At this point we have the ability to determine the `valid_loss` independently but have not taken out the protections in the `AutoBase` class, here: <https://github.com/Nixtla/neuralforecast/blob/main/neuralforecast/common/_base_auto.py#L67>

In the meantime what I have observed with the number of components is that it follows a classic bias-variance u shape behavior like the ones in this plot.

Another recomendation <@U04UKKD2F4K>, is to momentarily use the `alias`  parameter to distinguish the name columns between the results of different GMMs in the final `Y_hat_df` dataframe.

Hi <@U055ZP43Z4K>,

We use by default in the RNN-Based methods (RNN, GRU, LSTM, TCN,...) all the available history for the series at the time of the prediction y[:t].

If you want to reduce the inference time you can use the `input_size` , to make the inputs of fixed size y[t-input_size:t].

do we treat the problem of non stationary time series? in the library .
and what about this

Hey <@U04UTT23343>,

Thanks for using NeuralForecast.
Here is the hyperparameter tuning documentation where we explain it: <https://nixtla.github.io/neuralforecast/common.base_auto.html>

Some ideas to be able to compare other methods:
1. If the models are not Pytorch-Lightning classes, you will need to write your own hyperparameter optimization functions, that homogeneize the inputs and outputs of the models and its evaluation and use directly a library like <https://github.com/hyperopt/hyperopt|hyperopt>.
2. If the models are Pytorch-Lightning classes you might be able to feed them into the `BaseAuto` , and you could become an important contributor to NeuralForecast's model collection.
What are the models that you are looking forward to compare?

The other models are ML based (SVR, RF, etc). We would have liked to use mlforecast, but it doesnt currently support the inclusion of historical and future exogenous variables as far as I can tell.

Screenshot 2023-05-04 at 10.55.48 AM.png

One of the main advantages of NeuralForecast over MLForecast and alternatives is the automatic featurization, that avoids the hastle of feature engineering.
But in principle you should be able to feed the exogenous like in a classic XGB regression to model this problem:

As long as the rows of the forecast creation date match with your features, I think you should be able to use MLForecast.

On another note, the distinction between historic and future exogenous variables is fairly new innovation that very few forecasting models make. Which is another NeuralForecast advantage.

<@U038ZT9UDGA> do you know if there are any plans to add more 'featurization' to MLForecast? There are some pretty basic things that can extend it and gives good performance boosts.

Hey <@U056PNLC2TT> ,

If you have some ideas or feature requests add them here <https://github.com/Nixtla/mlforecast/issues|https://github.com/Nixtla/mlforecast/issues>

We are happy to include them

Hey Manuel,

None of our methods have this global-local model behavior.
But I don't believe that neural-prophet architecture is a good reference for accurate predictions.
<https://www.reddit.com/r/MachineLearning/comments/wqrw8x/d_fool_me_once_shame_on_you_fool_me_twice_shame/>

What has been your experience using the Neural Prophet?

Hi <@U038ZT9UDGA>! I have yet to try it but this Local-Global model idea might be interesting. At the moment I am trying hierarchical forecasting with neuralforecast and hierarchicalforecast but I can't get good results. So far the best combination I have found is this: 1) I forecast the time series at the highest level of the hierarchy with ARIMA. 2) I forecast all time series using NHITS with loss SMAPE. 3) I replace the forecasts made by NHITS for the highest level of the hierarchy with those made using ARIMA (basically I make a forecast mix where the highest level of the hierarchy uses ARIMA and the lowest level of the hierarchy uses NHITS). 4) I perform a Top-Down reconciliation using hierarchicalforecast. I also tried HINT but probabilistic losses (GMM, NBMM, PMM...) do not perform well for me. Considering the mix of local (ARIMA) and global (NHITS) models I use now, NeuralProphet's Local-Global model looked interesting.

Top Down reconciliation with ARIMA learns proportions to disaggregate ARIMA into bottom level series.
Depending on the version that you use 'average_proportions', 'proportion_averages' or 'forecasted_proportions', you could skip the NHITS model for the bottom.

Would you like to schedule a call to talk about your project?

<@U038ZT9UDGA> I need a global model at lower levels because some time series at lower levels of the hierarchy have a very short history and NHITS allows me to perform some kind of transfer learning to forecast short time series. Scheduling a call would be super nice!