Nixtla's Open Source Time Series Ecosystem.

Nixtla Community

Loved the aspect that you can specify the columns in the config instead of modifying the data, plus n_changepoints is very helpful! Great work!

Hi <@U055GTC3XUM>,

Thanks for trying out our libraries. Answering your questions:
1. Regarding distributed computation of the rest of the libraries:
    a. MLForecast supports distributed training, here is a tutorial: <https://nixtla.github.io/mlforecast/docs/quick_start_distributed.html>. 
    b. NeuralForecast allows for efficient GPU code execution, inherited from Pytorch Lightning. For the moment spark/distributed execution for NeuralForecast is not available.
    c. HierarchicalForecast by definition uses all the series from the hierarchy simultaneously. For the moment we are not parallelizing its operations. There is work to be done to improve its efficiency.
2. I don't believe that we have an evaluate function of the nature, would you be able to provide more information on it?

That was super quick, thanks!
1. Having the rest distributed is good enough, I don't expect reconciliation to be all that taxing. I'll have to check anyway. 
2. Sure! <https://www.sktime.net/en/latest/api_reference/auto_generated/sktime.forecasting.model_evaluation.evaluate.html#sktime.forecasting.model_evaluation.evaluate|https://www.sktime.net/en/latest/api_reference/auto_generated/sktime.forecasting.model_evaluation.evaluate.html#sktime.forecasting.model_evaluation.evaluate>
The above is quite handy as it allows me to build my own automl-type workflow without making any decisions for you about the forecasters, metrics, splitting etc. So given that the models are vecotrised, it internally predicts over the folds defined by the CV strategy and returns the results per fold, and you just have to loop over your config of models. Is there a way to do something similar for backtesting within the nixtla ecosystem? Seems like a fundamental functionality in the development of a forecast project.

Thanks for the pointer to the functionality, I will check it.

Hey <@U055GTC3XUM>! We have an evaluation function inspired by the `accuracy`  <https://fabletools.tidyverts.org/reference/accuracy.html|function> from the fable library.

You can pass the outputs of the `cross_validation`  method (to perform backtesting), and a list of metrics, and you will get the evaluation for each series and each window. In addition, the function can receive any dataframe (pandas, spark, dask, ray), so if you’re working in a distributed environment, you can evaluate your cross-validation windows quickly.


Here’s an example:

```from datasetsforecast.evaluation import accuracy
from datasetsforecast.losses import mae, mape

evaluation_df = accuracy(Y_cv_df, [mae, mape], agg_by=['unique_id', 'cutoff'])```
Feel free to let us know if you have any further questions. :)

Hey <@U055L8LSF6V>! Currently, we don’t have an automatic option for ensembling models. The good news is that we are working on that feature. In the meantime, this tutorial on ensemble statistical and neural methods can help. <https://nixtla.github.io/statsforecast/examples/statisticalneuralmethods.html>

Feel free to let us know if you have any more questions. :)

hey <@U056PNLC2TT>! Thanks for using MLForecast. Yes, you can pass your categorial columns as `category` type and the sklearn API of lightgbm (`LGMBRegressor`) will handle them by default. Also if you want to use the `unique_id`
as a static variable, you need to create a separate column. And to use it as a categorical feature you need to convert it to `category.`

Here’s an example (`unique_id`  already is of type `category` ):


```import lightgbm as lgb
from mlforecast import MLForecast
from mlforecast.utils import generate_daily_series


series = generate_daily_series(100, equal_ends=True, n_static_features=2, static_as_categorical=True)

series['my_id_col'] = series['unique_id'].copy()

mlf = MLForecast(
    models=lgb.LGBMRegressor(),
    freq='D',
    lags=[7],
)
mlf.fit(series, static_features=['my_id_col', 'static_0', 'static_1'])

preds = mlf.predict(12)```

image.png

This is what the series dataframe looks like:

Hey <@U0316L4HWQ7> awesome thanks so much. Any plans to make using the id column default behavior? Using a categorical id a long with scaling at the time series level (boxcox, standard, minmax) gives nice performance boosts!

hey <@U056PNLC2TT>! Could you help us open an issue to include that behavior? <https://github.com/Nixtla/mlforecast/issues/new/choose>

Thank you!

hey <@U0316L4HWQ7> sorry it took so long to get around to this but I have opened a couple of issues up. I also wrote a wrapper a little while back that does a lot of what I see gives good performance boosts. It only uses LightGBM but it adds some other features that make it a little easier to tune (in my experience) I pushed it to <https://github.com/tblume1992/Seance|github> if interested!

This is really cool, <@U056PNLC2TT>! wdyt <@U03UX9KR75W>? maybe we could include some of these features natively in mlforecast, would you like to contribute to that <@U056PNLC2TT>? :tada:

<@U03UX9KR75W> let me know if you want me to work on adding these. They are also pretty easy code snippets so we could just jump on a call and I could review them and you could add them. Whatever works best for you!

If you have time to open a PR that'd be great. I think you can create a new module with these features. Let us know if you need help, nbdev takes some time to getting used to

1. You know that cross validation for timeseries is slightly different right? Like <https://nixtla.github.io/statsforecast/examples/crossvalidation.html|this> 
2. Yes it benefits as much as normal machine learning. It gives you a clue how robust the model is to different folds

you mean by slightliy is the sampling for data flod shloud be next to each other right?

Thanks for the very detailed question. I think this is a great index for a tutorial that we should write.

In the mean time I would recommend you the following.
• <https://nixtla.github.io/statsforecast/examples/getting_started_complete.html|Follow this End to End Walkthrough >_<https://nixtla.github.io/statsforecast/examples/getting_started_complete.html|Model training, evaluation and selection for multiple time series.>_
Some brief answers to your points.
1.  a.) Nixtla does not support exploratory analysis per se. (Here is a <https://towardsdatascience.com/how-to-do-an-eda-for-time-series-cbb92b3b1913|tutorial> using pandas profiling)
TIP: Speak to your collegues from business and operations to find important categorical variables. For example, if there is a particular month where business ran a promotion you could try to create categortiable variables like promotion 1 or 0. The same for special things like covid.

1. b.) Short: don't use MAPE, maybe use MAE. 
    a. Long Read:
        i. <https://towardsdatascience.com/forecast-kpi-rmse-mae-mape-bias-cdc5703d242d|Forecast KPIs: RMSE, MAE, MAPE &amp; Bias> 
        ii. *<https://towardsdatascience.com/time-series-forecast-error-metrics-you-should-know-cc88b8c67f27|Time Series Forecast Error Metrics You Should Know>*
2. Yes, start with statasforecast. Then follow: <https://nixtla.github.io/statsforecast/examples/statisticalneuralmethods.html>

&gt;         i. Should I try it with all models and see the result with a crossval? Is it appropriate to tune these models? (I have not seen how to perform it in the documentation)
&gt; 1. 
Start with the auto models. These models find the best parameters for you.  AutoTheta, AutoMSTL, etc,

Hi Max,
Thanks for the answer :slightly_smiling_face:. It will help to start.
I confirm that a full tutorial for handling a time series forecasting project using all Nixtla tools (and others) would be great and helpful.

I did a first iteration with StatsForecast with Auto models, I got the first results (which seems good).
I quickly tried pandas-profiling, it is very simple for now.

I am going to follow this tuto now and see the results <https://nixtla.github.io/statsforecast/examples/statisticalneuralmethods.html>

hey <@U03LL877B8F>! Thanks for using the nixtlaverse. Complementing <@U035DR8HD6D>, here are a couple of ideas regarding your questions:


1. 
a. Yes, the best practice is to start the pipeline with a proper eda. A good approach to finding trends and seasonalities would be using the MSTL model (<https://nixtla.github.io/statsforecast/examples/multipleseasonalities.html>) to decompose the series and find relevant patterns. Also, you could use tsfeatures (<https://github.com/Nixtla/tsfeatures>) to extract relevant information about your series (such as sparsity, entropy, and autocorrelation strength, among others). Currently,  handling missing values is out of the scope of our libraries (we are working on that). But in this tutorial (<https://github.com/Nixtla/statsforecast/blob/main/experiments/bigquery/src/statsforecast-fugue-citibikes-trips.ipynb>) you can find an approach to filling them.
b. The mape metric usually is extremely hard to judge (<https://blog.blueyonder.com/mean-absolute-percentage-error-mape-has-served-its-duty-and-should-now-retire/>). Scaled metrics such as mase or rmsse might be a better option.
c. Usually, having more than one cross-validation window is a better practice. The exact number of windows depends on the quantity of your data, and your use case. If you have long time series and are willing to wait, we suggest using all the possible windows.

2.

a. Yes, that’s a good approach. StatsForecast contains the most simple models (Naive, and SeasonalNaive), so after creating a benchmark with those models, you should start building more complex ones.
b. In our experience, the automated models (AutoARIMA, AutoETS, AutoTheta, and AutoCES) produce good benchmarks and the tuning is performed inside them, so there is no need to tune them.
c. i. This is usually an empirical question. In our experience, an iterative process where you start with the most simple features and transformations (for example, adding Differences and lags) and then increase complexity leads to good results. Transformations such as scaling (MinMax, Standardize) and BoxCox are good options to test.
ii. Yes, the three libraries can handle external regressors. (StatsForecast through the AutoARIMA model). To evaluate if external data is relevant, perform cross-validation with and without that data and compare the models’ performances.

e. HierarchicalForecast assumes that you have a hierarchical structure (for example, if you want to predict national-level sales and state-level sales and want them to be coherent, that means that if you add up the forecasts at the state level you’ll get the national level).  The algorithms in the library are agnostic: you can use any algorithm to produce forecasts and then reconcile them using the library. Here’s an introduction to the topic: <https://nixtla.github.io/hierarchicalforecast/examples/introduction.html>.

Please let us know if you have any further questions.

Hi, Thanks again for these details. I will take into account all these comments.

I am trying to add exogenous data to AutoARIMA model and using a Crossval to evaluate if the perf is better than just using the target.
I checked this tuto, but it is done only by doing a train/test split : <https://nixtla.github.io/statsforecast/examples/exogenous.html#train-model>
But I can't find how to adapt it, there is not "X_df" in the StatsForecast.cross_validation(). Can you propose me something that I can use?

Another question regarding this statement: "If the future values of the exogenous regressors are not available, then they must be forecasted or the regressors need to be eliminated from the model. Without them, it is not possible to generate the forecast." My exogenous data is not known in the future, does it mean I have to forecast it before using any model? This could add bias?

Thanks again for your help.

I am trying to create features from tsfeatures, but I don't understand how the features created can be used within my dataset.
When I run it, I get the result below. What should I do with it?
Thank you very much.

Hey <@U03LL877B8F>! Usually those features are used to explore the characteristics of the time series to plan in advance what models would perform better and cluster them. Here’s a reference on the topic: <https://otexts.com/fpp3/features.html>.

Sorry <@U03LL877B8F>, I missed the previous questions. Here are some ideas:

• The `cross_validation`  method automatically handles the exgenous variables. So if you have more variables after the target column `y` , they will be considered exogenous variables and used by the models that allow it. Since, for each window, the exogenous variables of the future are available in cross-validation, their handling is done automatically.   
• Yes, an approach to use unknown exogenous variables is to forecast them separately and use the forecasted values to produce forecasts of the target variable. 

Hi <@U0316L4HWQ7>, Thank you again, I just learned new things about features :slightly_smiling_face:!

Regarding the `cross_validation` :
• Rolling window vs Expanding window : 
    ◦ Is there a way to choose one or the other? Or what is the best approach?
• This is the configuration I've used to test different models from statsforecast. I would like to do the same with `N-HITS` but I want it to be comparable (same number of windows. I am not sure how it will work because with `N-HITS` there must be a validation set to adjust the model and a test set? If I put `n_windows=100` with  N-HITS what will happen to val/test? Hope my question is clear.
```crossvalidation_df = sf.cross_validation(
    df=Y_ts,
    h=24,
    step_size=1,
    n_windows=100
  )```
I am going to start with NeuralForecast or MLForecast:

• Which one should I start with? (Remember my TS has around 180 months history). So not very long.
• For these 2 librairies should I create Features? Or it's just for ML Forecast?
• Feature Engineering: what is the best strategy? I read I could create a bunch of features and then select the best ones (lasso...), do you know any ressource that explain that with code?
• Do I need to normalize the data for N-HITS? And other models in NeuralForecast?
Thanks again for your valuable time.

hey <@U03LL877B8F>!

Yes, you can choose rolling window (default behavior) and expanding window (statsforecast and mlforecast) using `refit=False`  in `cross_validation`  (currently this option is only available for the Auto models in statsforecast).

If you set `n_windows=100` , those windows will be treated as a test set (the model will not use those values during training).

Usually, a good workflow starts with ststaforecast, mlforecast, and then neuralforecast. (less complex to more complex models). About features, neuralforecast does not need them, but you’ll need to specify them using mlforecast. The best strategy for feature engineering is to start with lags and simple transformations and then add more to see if the cross-validation signal improves.

It is always best practice to scale (or normalize) the data using global models (neuralforecast or mlforecast). The models included in the neuralforecast library can receive the `scaler_type`  argument to perform different strategies of scaling, here’s an example: <https://nixtla.github.io/neuralforecast/examples/longhorizon_probabilistic.html>

Hi <@U0316L4HWQ7>, thank you for your feedback.
I am following your advice and process.

Regarding the feature engineering with cross-val, let say I start with lag 1, check the score, then I add lag 2, check score, then... Until lag 12... And select best features from scores I have seen.
Then I start the same process with window features... It will take forever.

Is there a way of automating this process? Or this is how it should be done?

Thank you.

Hi!
An important question first is to understand why the data is missing:
• In case of retail, missing data very often means that there where no sales. So, missing data should be replaced with 0s.
• Otherwise you might want to fill the missing data, doing some interpolation. 
Could you give us more context?

Hi <@U055ZP43Z4K> Here's more information on prediction intervals. <https://otexts.com/fpp3/perspective.html>

hey <@U055ZP43Z4K>! If you’re referring to the NHITS paper, here’s the link: <https://arxiv.org/abs/2201.12886>.

Hey <@U040AMG6N69>! The underlying assumption behind some statistical models (such as ARIMA) is that the data they are modeling follows a normal distribution. To compute CI, the mean and the variance of such normal random variable are calculated for each timestamp in the horizon. So, you could compute those parameters (the mean corresponds to the point forecasts and you can calculate the standard deviation by computing a CI with a level of `68.27` ). After that, you can compute the parameters of the average series (using the fact that the sum of normally distributed random variables is also a normal random variable).

<@U0316L4HWQ7>  68.27 because that's one standard deviation away from the mean? And then in that case the lower and the upper bounds of my CI is just the STDev and the mean is the point forecast so I end up let's say with 100 means and 100 STDevs (assuming I have 100 time series) at each point in the horizon and I can assume these means and STDevs are normally distributed so I take average of my averages at each point in time but then can I take the average of my STDevs to build the global CI? I think I'm getting close here but not totally there yet. Need to better understand how to calculate the STDev for the final calculation of the global CI.

hey <@U040AMG6N69>! Yes, that’s the intuition behind 68.27. Regarding the distribution of the average, if you have 100 series each of one with mean `m_i` , and standard deviation `sigma_i`  then the distribution of the average will have mean `(m_1 + m_2 + ... + m_100)/100`  and standard deviation `(sigma^2_1 + sigma^2_2 + ... + sigma^2_100) ^ (1/2) / 100`  (the root of the sum of the variances divided by 100).

hey <@U055ZP43Z4K>! Thanks for using NeuralForecast.

The `val_size`  argument of the `cross_validation`  method in neuralforecast is used by the `Auto`  models to perform hyperparameter optimization. The `test_size`  and `n_windows`  are interchangeable and work the same as in StatsForecast and MLForecast.

so if i am doing cross validation for non AUTOModels i dont need to valdation size right ?

Yes, exactly, if you’re not using the auto models you can skip defining `val_size`  :slightly_smiling_face:

Hi Isaac, we are working on ways to optimize hierachicalforecast and also ways to distribute the reconciliation strategies.

I'm going to DM you to explore alternatives.

hey <@U055ZP43Z4K>! Could you explain more about what you refer to by historical training?

i am sorry i meant historical forcast like in darts

hey <@U055ZP43Z4K>! You can recover the historical forecasts using the `predict_rolled`  method. Here’s the reference: <https://nixtla.github.io/neuralforecast/core.html#neuralforecast.predict_rolled>

We deprecate `predict_rolled`  in favor of `predict_insample` , but we must update the documentation accordingly. :)

hey <@U055L8LSF6V>! Thanks for using mlforecast. You can define a specific `Transform`  class to standardize individual time series before fitting and predicting (this accounts for different scales). But you can also specify a `Transform` class to standardize the series globally if you prefer. Here’s an example (standardizing each individual series): <https://nixtla.github.io/mlforecast/docs/end_to_end_walkthrough.html#target-transformations>.

Regarding the `cross_validation`  method, the splits are done before fitting the models. Therefore the transformations are only applied to the training data of the specific window.

Please let us know if something else needs to be clarified. :)

hey <@U056QNCB7SS>! If I understand correctly, after applying `aggregate`  the 2nd ts has zero values from January 2019 to April 2019? I think this is an unexpected behavior; those zeros will probably negatively impact the model’s performance. Could you help us open an issue on the github repo to investigate the problem? Here’s the link: <https://github.com/Nixtla/hierarchicalforecast/issues/new>

Thank you!

Yes.  And if you try and remove the zero's, you're unable to reconcile, because there's a shape issue with the fitted dataset

<https://github.com/Nixtla/hierarchicalforecast/issues/189>

thank you <@U056QNCB7SS>! We’ve opened a PR fixing the issue (<https://github.com/Nixtla/hierarchicalforecast/pull/190>). We will merge it into the main branch and publish a new version soon. In the meantime, you can test the changes installing from that branch: `pip install git+<https://github.com/nixtla/hierarchicalforecast.git@189-aggregate-adds-leading-zeros-to-series-with-different-dates>` .

Please let us know if this helps or if something else needs to be clarified. :)

Cool cool, I'll take a look.  Did you test whether `reconcile` still works as expected?

`aggregate` works as expected! But `reconcile` fails.  I added a reproducible example to my github issue.

Hey <@U056QNCB7SS>! Thanks for letting us know about the problem, we’ll be working on it :slightly_smiling_face:

Good to know.  Let me know if I can help :slightly_smiling_face:

<@U0316L4HWQ7> when do you think you'll push the fixes?

hey <@U056QNCB7SS>! We’ve pushed the fixes into the main branch. We will release a new version in the following days :)

Woohoo! Excited to try it out. I also saw the conformal predictions in statsforecast coming out soon :eyes:

Nice! Feel free to let us know if something breaks or if you need any help. Yeah! We are excited about conformal prediction in stats, we have to finish some details, but in the next weeks should be released :tada:

Any update on when the new hierarchicalforecast package will be released?

hey <@U055L8LSF6V>! XGBoost is generally faster due to the underlying gradient calculation. You could reduce the complexity of the hyperparameters of the random forest model, reducing `max_depth` , and `n_estimators` .

hey <@U040AMG6N69>! That might be a good approach, but in the end, it depends on the data. here’s an interesting reference on the topic: <https://otexts.com/fpp3/complexseasonality.html#dynamic-harmonic-regression-with-multiple-seasonal-periods>

Also, you could use the MSTL model to capture the seasonal components of your time series.

<@U0316L4HWQ7>  much appreciated! Thanks!

hey <@U055ZP43Z4K>! here’s a good thread on the topic: <https://github.com/Nixtla/neuralforecast/issues/568>

Yes, it is possible to add external features such as weather temperature or trend, seasonality of series that you will forecast. The `MLForecast` class in `mlforecast` framework supports exogenous variables and static covariates. These can be included as additional columns in the pandas dataframe that is passed to the `fit` and `predict` methods. You can define these additional features by creating additional columns in the pandas dataframe that is passed to the `fit` and `predict` methods.
*See*
• <https://nixtla.github.io/mlforecast/docs/end_to_end_walkthrough.html|https://nixtla.github.io/mlforecast/docs/end_to_end_walkthrough.html>

Thank you for your interest. I checked this link but i couldn't see the anything about adding specific features such as trend, seasonality of series.  I add lag features, date features at the below picture. However when i try to add trend of this serie, it cannot be added. What should i do now ?
Note: I decomposed the serie and obtained trend component at the above of my notebook.

Hi <@U057ZJSJVPD>, you are right. I was under the impression we included static and dynamic covariates in that tutorial. Please check this link. You can find an example on hoe to add external features
<https://nixtla.github.io/mlforecast/forecast.html#dynamic-features>

Hi <@U035DR8HD6D>, Thank you! I have one more question about it. You're saying that we can add the features such as trend, temperature, sensor data etc. as dynamic features. However, when we forecast next steps (future), i have again dynamic features and i want to see the effect of the dynamic features at the end of the forecast. Here is an example about it:

Let's say, i have demand data daily. Then i want to forecast demand for next 2 steps (future). And i have also dynamic feature like flag that contains 1's and 0's. Then, i want to add the effect of the this flag for the forecast. I also add example picture above.

In a summary, I know what happened in the past when flags were 1 and 0 and how many demands were received. Since I know that it has an effect on demand and I want to see this effect in my forecasts in the future demands, that's why, I should use this flag feature when forecasting. Can i do this with MLForecast?

Thank you for your interest.

Hello Rachel, I recommend this post, maybe it can help a little more <https://www.linkedin.com/pulse/autoarima-con-statsforecast-naren-castellon?utm_source=share&amp;utm_medium=member_android&amp;utm_campaign=share_via|https://www.linkedin.com/pulse/autoarima-con-statsforecast-naren-castellon?utm_source=share&amp;utm_medium=member_android&amp;utm_campaign=share_via>

if my data random and it has one reguler peak  at each day any recommendation for how can i start with it ?

I was having the same doubts with NBEATSx, but I solved it with a simple experiment.

I literally gave a slightly modified version of the prediction (yhat) as exogenous feature, and the model was able to capture it. So maybe try that just to check that it can do that.

Like you purposefully introduced data leakage to see if the model can use it to improve? I'll give it a try.

Precisely! My logic was - If exogenous variables are truly informative for the model, then it should be able to capture the leak and reproduce the results.

So I did just that, and after some tuning (namely, improving the size of the NN) it worked wonders

Thanks Narem, this is great. We have been working on the GitHub guide and will contact you tomorrow. Sorry for not answering before. We were traveling. 

hey <@U055ZP43Z4K>! We haven’t tested the library on non-nvidia gpus, but pytorch and pytorch-lightning support AMD gpus so it should work :)

Ciao Pietro, was a nice talk, had fun listening in :muscle: :muscle: 

<@U03DMTQR013>, that is amazing. Thank you very much.

The example <https://github.com/Nixtla/mlforecast/issues/124#issue-1683960264|here> is a good way to do it atm

hey <@U053D3591PC>! Currently you can use mlforecast on databricks, here’s an example: <https://nixtla.github.io/mlforecast/distributed.forecast.html#spark>.

Regarding neuralforecast, could you help us open an issue to work on that feature: <https://github.com/Nixtla/neuralforecast/issues/new?assignees=&amp;labels=&amp;projects=&amp;template=feature_request.md&amp;title=>.

maybe the `cross_validation` method from the core neuralforecast can help?

it basically trains your model up to a point then forecast for a given number of horizons in the future, which could be week 1, 2, .., n

That's certainly a possible workaround, thanks! For it to work properly I assume I'd have to append the last n observations from y_train to y_test and run cross_validation.

That's a fair workaround. I tend to leave out some of the dataset/y_test for out-of-sample testing (especially the most recent past), just to check for overfitting

How would this work for SimpleExponentialSmoothingOptimized though? This has no `forward` method. I'd expect to be able to find the optimized alpha value on the training set and then call forward on the test set. Being able to use y_true in a rolling way is a big requirement for me

I guess it does just that. I wasn't familiar with the model, but was able to whip up some quick example to show it in action: <https://gist.github.com/rcslight/81651ff9d8e372026100ebd32a6789e8|github gist.>

from what I get:
• it cuts off the training data at 1958-11-30, fixing alpha
• then uses the following y_hat as the rolling weights

Thank you so much for helping! The issue is that in our use case we might have access to y_true so we want to split experiments in cases where we use y_hat in a rolling way (as in your gist) and cases where we replace the oldest y_hat with y_true. I _think_ forward does this (although it isn't really documented what it does...) but this model does not have it :confused:

I guess you're right. Maybe a loop over fit/predict would work, where you choose whether to paste y_hat or y_true?

Yeah, I can make it work with a few ugly workarounds or looping. That's totally ok. Since the package already does so much I was implicitly expecting it to do eveyrhting.

I can open an issue on Git to support this because I think it's a common enough use case

perhaps there's a place for a `rolling` flag in `cross_validation` to do just that

it means making predictions on the same time series data that was used for training the model.