Nixtla's Open Source Time Series Ecosystem.

Nixtla Community

I went back to this and fixed the issue, here is a PR: <https://github.com/Nixtla/nixtla/pull/44> I will keep working on reproducing the colab notebook and let you know how it goes (and possibly ask more questions if I get stuck again) :simple_smile:

Thank you very much pietro. We have been very busy
Planning a new release and haven't been able to check this but <@U0316L4HWQ7> will take a look at it next week. 

<@U0316L4HWQ7> is already working on that blogpost and we will send a draft to you in the next 2 days.

Awesome, and thank you for creating this project / crafting the blog post! We've been very excited to see the ways that you are stretching Ray, and for such practical use cases. :smile::raised_hands:

Thanks, <@U03DDFKMT0X>! Please let us know if you would like help testing the API or if you would like us to help you with a specific use case

sounds good - i'll reach out via DM shortly

Thanks for the support :slightly_smiling_face:

Hi all, good to be here, greetings to <@U035DR8HD6D>, <@U031KHZH06B> and the Nixtla team

We saw your call to action to the conformal predictions, is there a torch or pointers to the technique?

We are also curious about your experience with the transferability of the MQN-HiTS

<@U038ZT9UDGA> yes this is the place to go for conformal prediction. <https://github.com/valeman/awesome-conformal-prediction> have not spun MQN-HiTS yet but certainly on my plan of things to do.

Thanks for the pointer, we will take a look

There is conformal prediction slack as well <https://join.slack.com/t/awesomeconformalpred/shared_invite/zt-193zf5qwx-uW7z9RWlB~_cUQq6iVIPyg|https://app.slack.com/client/T02S1RLPGJE >a few authors of time series conformal papers there already. Plus forecasting channel, most welcome to join and collaborate.

Yes, it would be great combo because generally they produce too narrow (overconfident) prediction intervals

I don't follow too much Data Robot, we have not had very good experiences with it, last time we checked they had XGBoost and RandomForest wrappers for forecasting.
Thanks for the update on what they are working on.

Thanks a lot for the great article. We are really humbled :slightly_smiling_face:

<@U035DR8HD6D> I know the feeling :sweat_smile: the team is doing awesome. The article is unbiased reflection on where the open source forecasting is and Nixtla is really great. My team have used a lot of Nixtla tech to create beautiful forecasting models for the business and more will be added.

Thanks for your question. We will come back to you soon. 

Hello <@U035DR8HD6D>, do you have any updates on this? Thank you, and thank you for this amazing package!

<@U0316L4HWQ7>, could you provide an answer?

Hi, just as a follow up, I have found a quick way of doing this through the AutoARIMAProphet class! Thank you

Hi <@U03LHHAF6SX>! Thank you for using `StatsForecast` . For the moment, those parameters can’t be modified (we are working on this). There is a workaround though: the `auto_arima`  function uses `auto_arima_f` . So you can create your own `auto_arima`  function using:

```from statsforecast.arima import auto_arima_f, forecast_arima

def auto_arima(X: np.ndarray, h: int, future_xreg=None, season_length: int = 1,
               approximation: bool = False, level: Optional[Tuple[int]] = None) -&gt; np.ndarray:
    y = X[:, 0] if X.ndim == 2 else X
    xreg = X[:, 1:] if (X.ndim == 2 and X.shape[1] &gt; 1) else None
    mod = auto_arima_f(
        y,
        xreg=xreg,
        period=season_length,
        approximation=approximation,
        allowmean=False, allowdrift=False, #not implemented yet
        ##EXTRA PARAMETERS HERE
        max_p=1,
        max_q=1
    )
    fcst = forecast_arima(mod, h, xreg=future_xreg, level=level)
    if level is None:
        return fcst['mean']
    return {
        'mean': fcst['mean'],
        **{f'lo-{l}': fcst['lower'][f'{l}%'] for l in reversed(level)},
        **{f'hi-{l}': fcst['upper'][f'{l}%'] for l in level},
    }```
Then, you can pass your `auto_arima`  to the `models`  argument.

hey, sorry for jumping on this thread. I was wondering if there’s a way to extract the p,q values picked by the model after fitting it? (using `StatsForecast`)

Sure <@U03DDFKMT0X> <https://www.reddit.com/r/MachineLearning/comments/wqrw8x/d_fool_me_once_shame_on_you_fool_me_twice_shame/|https://www.reddit.com/r/MachineLearning/comments/wqrw8x/d_fool_me_once_shame_on_you_fool_me_twice_shame/>

Definitely interesting! We've had a lot of success with `NeuralProphet` but always happy to try out other libraries :slightly_smiling_face: do you have a tutorial recommendation in `statsforecast` for multi-step prediction with exogenous lagged + future regressors?

would be super interesting to see the results between `statsforecast` and `NeuralProphet` when more extensive hyperparam optimization is applied, this looks like a single architecture was used throughout: <https://github.com/Nixtla/statsforecast/blob/main/experiments/neuralprophet/src/neuralprophet.py> ... FWIW, we've found we need around ~70 NN architecture searches followed by about the same number of LR tuning runs, definitely not ideal but not unlike other NN approaches. fortunately the auto LR finder works reasonably well which makes the arch searches a lot more reasonable.

we'll try to bring in the example from <https://github.com/Nixtla/statsforecast/blob/main/experiments/arima_xreg/src/statsforecast.py> this weekend and report back on any interesting findings we come across!

<@U03DDFKMT0X> please share will be interesting to see the results 

okay, we still need to do some more testing but so far we're finding that a properly tuned model from `NeuralProphet` consistently outperforms both `AutoARIMA` and `ETS` from `statsforecast` on a wide range of power system forecasting tasks. i can't comment on whether this extends to other time series baseline datasets but i suspect the ERCOT baseline is very similar to our datasets. obviously the tuning time to get decent results is a disadvantage relative to `AutoARIMA` and `ETS` but it seems the same tradeoff exists between `NeuralProphet` and `Nbeats / N-HiTS` in that it's relatively quick to spin up decent models in `NeuralProphet` whereas `Nbeats / N-HiTS` requires a much more extensive hyperparmeter sweep to get comparable results.

we're still pretty new to the `nixtla` ecosystem of open source forecasting tools though, so i'm sure there are many things we're doing wrong, happy to chat about opportunities for improving our integrations with either `statsforecast` or `neuralforecast`

FWIW, we may replace `RandomForestRegressor` from `sklearn` as our baseline in favour of `AutoARIMA` from `statsforecast` - works great and super efficient

Hi Sean, currently thats not possible with statsforecast. We are, however, working on that feature.
There are a couple of options meanwhile
• Use Ray Clusters for distributed computing. Here is a <https://www.anyscale.com/blog/how-nixtla-uses-ray-to-accurately-predict-more-than-a-million-time-series|blog post>.
• Try using other models from neuralforecast than can be “updated”. Here is an example that can give you an idea: <https://colab.research.google.com/drive/1uFCO2UBpH-5l2fk3KmxfU0oupsOC6v2n>

I see that the second notebook you sent is on transfer learning. There might have been a misunderstanding, I am not trying to predict a different time series, just the continuation of the time series as new data comes in - precisely the use case for the ARIMA model.

How are the statsforecasts models used when actually featuring in a pipeline? Do they have to be retrained by scratch after every observation?

Thank you again for your time and help

Sorry, maybe I wasn’t very clear. The case of predicting a time series with a pre-trained model on that same time series could be understood as a very specific case of transfer learning.
In other words, you could follow the Colab and use the model you trained on your data to predict your own and same data. Hopefully, you will achieve your practical goals. Namely: saving time on new training without compromising  too much accuracy.

Regarding `statsforecast`: currently yo do have to retrain every time you want to update the model with new data. That doesn’t necessarily mean you have too retrain with every new observation.
Example: you want to predict the next 48 hours every 24 hours although you get new info every hour. In that case you would retrain from scratch every 24 hours, say at noon.

In an oversimplification, how people tend to deploy this models into production is by orchestrating  workflows or pipelines that:
1. Extract data, do some wrangling and load to memory every so and so or given a specific event
2. Run the algorithms. Sometimes people run different models and then choose the best one bases on defined error metrics. 
3. Output the prediction somewhere
For simple pipelines we have seen people use tools like cron on servers on the cloud.
For more complicated pipelines you can use things like Prefect, Airflow, Ray, Mlflow and even run the computation on clusters of different machines. If you use spot instances, this process is actually quite cheap and fast.

I’m sure you knew most of this stuff but wanted to be as clear as possible.

If you need help with your particular implementation we could schedule a call or something.

Hello Max, thank you so much for the thorough response. I think that everything is clear now! Have a great weekend

Welcome <@U03NX76Q6CB>! Happy to see you here.

&gt; Hello everyone :slightly_smiling_face: !

image.png

Hi <@U03NX76Q6CB>! Thanks for using `tsfeatures` . :D

The argument `features`  receives a list of functions, so it is necessary to import `acf_features` . Also, the `freq`  argument should be an integer instead of a string. Since you are using `30T`  data, `freq=48` might be a good option (to reflect daily seasonality).

With these changes the code would be,

```from tsfeatures import acf_features

tsfeatures(panel, freq=48, features=[acf_features])```
Please let me know if you have any other questions. Happy to help. :)

<!channel>: Yesterday we released our new :crown:HierarchicalForecast library.  :fire:

Show some support by giving us a :star2: <https://lnkd.in/eiCSKaPi|https://lnkd.in/eiCSKaPi>. 

Now you can reconcile forecasts for hierarchical problems in a simple way using statistical approaches such as Bottom Up, Top Down, Middle Out, Minimum Trace, and Empirical Risk Minimization. Moreover, you can achieve state-of-the-art results. See Benchmarks: <https://lnkd.in/e8p3ntbM|https://lnkd.in/e8p3ntbM>  :rocket::rocket::rocket:

Hierarchical :crown: forecasting is important where time series data can be grouped or aggregated at various levels.  Classical examples include aggregation of sales from product level to brand levels or geographical aggregations from zip code to country. Since these categories are nested within the larger group categories, the collection of time series is said to follow a hierarchical structure.   :fire::fire::fire:

Hi! Happy to help with anything we can :slightly_smiling_face:

Looks like this community makes some really fun things.

Thanks! You might be interested in this thing particularly: <https://colab.research.google.com/github/Nixtla/neuralforecast/blob/main/examples/mqnhits.ipynb>

Given your interest in probabilistic forecasting

data2.png

Hi <@U03RM1M35UL>! For the moment, you can’t use integer datestamps to produce forecasts. If you have a map for each int to a monthly date, then you will be able to produce forecasts.

If don’t have a real mapping you could obviously use one artificial one.
e.g.
1 -&gt; 01/01/1900
2 -&gt; 02/01/1900
…

<@U035DR8HD6D> <@U0316L4HWQ7> yes, I have the mapping of each int to the given datestamp, I convert the datestamp to the numbers so that its structure will be the same as that of M4 but still it’s not working.

I'm very sorry this is not working. Could you share a little bit more info on the error or maybe even share some code so we can reproduce the error ?

<@U035DR8HD6D>  training is not working it is stopping in between although the data is correct, I have shared the data and the notebook with u in DM please check

Hello, how are you doing? I'm trying to adapt my work to auto_arima from the current use of pmdarima that is way too slow for the work I'm doing and it seems to work faster indeed. However I'm trying to see how to calculate the aic value of the model and for that I need the order (p and q), is there anything available to get this data

Yes, I'm working with that, thanks! But I was trying to get hold of the order of the model calculated and the AIC value

We are currently working on the new sklearn-like classes with fit/predict methods that store the parameters

For the moment there is a tradeoff between speed and available information within the models. We created the options to run without giving the parameters back which is really fast
vs running with the parameters as output too which is a bit slower

Be ready to test it!!! My current models are taking 10 days to run....

We have a pending blog post on how to run 10^6 series with ETS under 2 minutes

Quick question <@U03TCNSTPB3>
Are you trying to use AIC as validation tool?

Would it not be better to use a cross validation measurement of error?

AIC and leave-one-out cross validation have known connections since the 80s/90s

I'm using both Akaike and MAE as measurements, as the models I'm using run with over 300 exogenous variables, and trying to see if the outcome is related to aic or mae..,

as in this case it seems that aic is not a good predictor of the model

Might be a good idea to try LASSO regularization on those exogenous
Or Ridge regression

So many exogenous variables could give you some autocorrelation problems

yes, they are creating problems, but that is part of the exploration as I'm trying to cluster the temporal series before and check if they convey additional information that may be useful for the models

StatsForecast cross_validation method can help to create predictions for rolling windows with which you might be able to get better measurements of prediction performance

I'll try that, thanks for the suggestion!

Have you tried PCA before on top of the exogenous?

not yet... its on the list to measure differences between methods.

Here is a <https://colab.research.google.com/github/nixtla/statsforecast/blob/main/examples/Getting_Started_with_Auto_Arima_and_ETS.ipynb|AutoARIMA/ETS tutorial>.