Nixtla's Open Source Time Series Ecosystem.

Nixtla Community

Hey <@U04QSAXB727>! That would be an amazing enhancement. The function that transforms the dataframe into numpy is this: <https://github.com/Nixtla/statsforecast/blob/main/statsforecast/core.py#L407>.

Hey, I had a follow up question, I'm currently trying to use your synthetic data generator function (<https://github.com/Nixtla/statsforecast/blob/606d983f0eacf89594707923b194aa2b89cfdc4c/statsforecast/utils.py#L16|https://github.com/Nixtla/statsforecast/blob/main/statsforecast/utils.py>). In order to use it as polars DataFrame I would need to recreate the steps from scratch due to the categorical dtypes (I have no clue why that's such a pain in the @$$ to deal with). Prior to doing that I wanted to ask you what's the point of having those values as categorical?

hey <@U04QSAXB727>! It is just for efficiency purposes when dealing with experiments that have thousands of time series, it’s more memory efficient to deal with categorical instead of object with such a large amount of data

maybe we could add a new argument to `generate_series` controlling if the conversion is required, something like `uid_to_categorical=True`

okay rebuilt it, also got some time shaved of the generation:
```before:
________________________________________________________
Executed in    3.93 secs    fish           external
   usr time    4.68 secs    0.06 millis    4.68 secs
   sys time    1.44 secs    1.03 millis    1.43 secs

after:
________________________________________________________
Executed in  833.23 millis    fish           external
   usr time    2.23 secs      0.07 millis    2.23 secs
   sys time    0.50 secs      1.20 millis    0.50 secs```

This is the most useless improvement :rolling_on_the_floor_laughing:

<@U04QSAXB727> we think its great. Thanks.

[PR Update] After two months of persistent effort, I have successfully developed a versatile DataFrame conversion solution for both Polars and Pandas. This solution is compatible with any structured two-dimensional data, provided that the output can be consolidated into named Numpy arrays.

:white_check_mark: All local tests have been passed, Github Actions required `util.py` to be modified, as I have added Polars test.

NOTE:
• Have not yet implemented engine into `_StatsForecast` and/or `StatsForecast` class for I/O to match.
• `core.ipynb` has `util.py`'s `generate_series` function so that code can actually run. Will be removed in future. 
More info:
<https://github.com/Nixtla/statsforecast/pull/448#issuecomment-1537431035>

We are going to write a brief post to communicate the new feature. Which accounts should we  mention to thank you.

<@U035DR8HD6D> Thank you, I'm glad to help.
I just implemented a change where if input dataframe is polars or pandas `StatsForecast`  should run without any issues.
TODO:
• Modify `plot` staticmethod to work both with polars and pandas
• Implement I/O matching, at current moment there is variable for that `self.engine`
    ◦ At current moment it is: 
    ◦ polars in and pandas out
    ◦ pandas in and pandas out

<@U035DR8HD6D>
All the local tests have been passed on Polars and Pandas using modified `generate_series`.

More information:
<https://github.com/Nixtla/statsforecast/pull/448#issuecomment-1546411593>

<@U0316L4HWQ7> could I ask you to please check the PR and let me know if I've missed anything. From my end everything worked smoothly.

core_polars_test.ipynb

<@U0316L4HWQ7> here's the file with all the tests done with Polars.
NOTE: `groupby` doesn't have sort param in polars, hence, have to chain `.sort('unique_id')`
P.S. Latest PR is up with bug fixes

Hey, there is a `_parse_ds_type` bug on `main` where in `int` datestamps are converted into datetime in certain cases, due to the check failure; specifically `issubclass(df["ds"].dtype.type, np.integer)` , which can be checked much better with kind,
`np.array().dtype.kind in ["i", "f"]` , where `i` stands for int and `f` stands for float.
Implemented this change in my PR: <https://github.com/Nixtla/statsforecast/pull/448>

hey <@U04R9CEKAF5>! Thanks for asking. What library are you referring to? :)

hey <@U04R9CEKAF5>! Currently, mlforecast doesn’t have a custom save and load behavior. Maybe you could try storing the fitted mlforecast class using pickle and after that call `predict` with `new_data` .

<@U04R9CEKAF5>, if you need that, we are happy to work on that.
Would you mind opening a request on <https://github.com/Nixtla/mlforecast/issues/new?assignees=&amp;labels=&amp;template=feature_request.md|our Github>?

I actually just reused the optimised parameters and tested the performance by refitting the model with the same (optimised) parameters. However, I stumbled upon one problem with this. The PredictionIntervals seem to be recalibrating as well which I don't want to do on the test set. Is there a way to move them from training to test somehow? Or abandoning them completely and just using levels in predict as another hyperparameter would do?

No problem, thanks for reading my tweets haha (if you are not following me yet, I cordially invite to do so!)

Hi Merlin,

Regarding your first question, I think that's a limitation inherent to the model and not just to StatsForecast's implementation. If you don't have the future values of the exogenous regressors, then you can forecast them first. This adds another layer of uncertainty, but might work with your data (you need to test it). I've done it before with prices when forecasting for a retailer and got acceptable results.

image.png

Regarding your second question, you can use exogenous regressors in crossvalidation. See the API for reference.
<https://nixtla.github.io/statsforecast/core.html#statsforecast.cross_validation>

Okay, thank you for taking the time to respond :slightly_smiling_face:

Future exogenous values are just taken from the training set for cross-validation. For pre-known future features like "is_weekend", 'is_holiday" this is a valid approach, but in case they are unknown, there is no separate prediction (cf. <https://github.com/Nixtla/statsforecast/blob/main/statsforecast/core.py#L289>). Maybe you could outline this to others in a tutorial.

I suppose for more sophisticated predictions, I will be better off using neuralforcast then?

hey <@U04RVB0EJRJ>! Currently, statsforecast doesn’t support the distinction between historical exogenous variables and future exogenous variables (available at the time of the prediction). For that kind of modeling, we suggest using neuralforecast :)

Is there a way to use exogenous variables with the AutoLSTM etc. in neuralforecast?

hey <@U04RVB0EJRJ>! Yes, here’s an example of how to include exogenous variables: <https://nixtla.github.io/neuralforecast/examples/exogenous_variables.html>

For the auto versions, you have to include the `futr_exog_list` , `hist_exog_list` , and `stat_exog_list`  in your hyperparameter grid

hey <@U03L39VLJBA>! The easiest way to recover R’s simple exponential smoothing (ses) is to call `AutoETS` with only additive errors (`model='ANN'`). Here’s an example of how to recover the parameters (the first one corresponds to `alpha`)

That’s great many thanks <@U0316L4HWQ7> 

Hi <@U0316L4HWQ7> I am trying to replicate R result for this real time series [5,6,3,2,6,3,9] the alpha produced by HoltWinters from R is 6.610696e-05 and this is the result of fit in R [5.0000, 5.0001, 4.9999, 4.9997, 4.9998, 4.9997, 4.9999]. Using the suggested code I am getting a different alpha 1.00000000e-04 and I guess different result. Do you reckon exact replication is possible using one of Nixtla models? Also how does one obtain predictions in sample for the above example? I have tried predict_in_sample() that i used for SimpleExponentialSmoothingOptimized but for AutoETS it did not seem to work

<@U03EP6RE8JU>: could you try to replicate?

<@U03EP6RE8JU>: while you figure this out, could yo create a github issue so we have track of this.

<@U03EP6RE8JU> <@U0316L4HWQ7> <@U035DR8HD6D> I wonder if there is a way to extract alpha from SES Optimized similar to what Fede shared  for ANN model. Getting alpha back from all exponential smoothing is very important. Our devs say currently there doesn’t seem that it is able to return alpha at the moment.

Yes, that's correct. And I agree. Getting the alpha is an important feature worth including in the exponential smoothing models.

Hi <@U03L39VLJBA>

I started to review this issue, but I haven't been able to replicate your result.

Using the forecast R package v 8.16, the Holt-Winters function hw( ) returns an error (it says that frequency should be greater than one) and the Holt function Holt( ) returns alpha = 0.6075.

Hence,
- Is the frequency of the time series 1?
- What function did you use to obtain the values you mentioned before?

Hi <@U03EP6RE8JU> HoltWinters in R, frequency 52, both beta and gamma set to False

Our devs used this setup which should reproduce the numbers 

This one <https://www.rdocumentation.org/packages/stats/versions/3.6.2/topics/HoltWinters|https://www.rdocumentation.org/packages/stats/versions/3.6.2/topics/HoltWinters>

If difficult, don’t spend time on it please, we put Nixtla one as replacement for now but would be great to have alpha returned added to future functionality releases if possible please 

Ok, thank you. I was using the Holt-Winters function from the forecast package, but now I see that it should the Holt-Winters function from stats.

And yes, we're looking into returning the alpha because we agree that it's an important functionality to have.

Thanks for the comment. Cc <@U03UX9KR75W> 

Hey <@U04U3RSDG4E>. I think that if you have any null values in your target you get an error when trying to run the preprocess because all transformations propagate the null values, were you able to run it with missing values in the middle?

Hi <@U03UX9KR75W>. thanks for the reply. the target does not have null values. the issue is with `rolling_std` when we have a set of consecutive zeros larger than window size in target. the root cause of issue seems to be  `_rolling_std`  in `window_ops.rolling` where it generates large negative numbers in above situation and these negative numbers are then converted to NAN. attached is an example.
```data = pd.DataFrame({
    'date': pd.date_range(start='2019-01-01', end='2020-12-31', freq='MS'),
    'sprid': 1.,
    'target': [1., 2., 0., 4., 0., 0., 0., 0., 9., 10., 11., 12] * 2
})

models = [lgb.LGBMRegressor(**{})]
fcst = MLForecast(
    models=models,
    freq='MS',
    lags=[1],
    lag_transforms={
        1: [(rolling_std, 3)]
    }
)

preprocessed_df = fcst.preprocess(data, id_col='sprid', time_col='date', target_col='target', dropna=False)
print(preprocessed_df)

## check _rolling_std
from window_ops.rolling import  _rolling_std
a = np.array([1, 2, 0, 4, 0, 0, 0, 0, 9, 10, 11, 12] * 2)
print(_rolling_std(a, 3))```

Thanks for the example! I think a warning for these cases is definitely useful. Will add it soon

additionally, `rolling_std` should generate zeros instead of NAN for these cases.

Hi  <@U04U3RSDG4E>! Happy to work on that.
Would you mind opening an issue for that on GH.
<https://github.com/Nixtla/statsforecast/issues/new?assignees=&amp;labels=enhancement&amp;template=feature-request.yml&amp;title=%5B%3CStatsForecast+component%3A+Models%7CCore%7Cetc...%3E%5D+|Here> is the link.

Hey <@U04U3RSDG4E>, we just pushed window-ops 0.0.14 which should fix this issue with the rolling_std

have you tried calling `cross_validate(fitted=True)`, then `cross_validation_fitted_values()` ?

Hi Sam, But there is no function *cross_validate* for spark FugueBackend, may you please share some snippet

Thanks <@U04UECRAUN7>, currently the dist engines don't support returning the fitted values.
If that is something you need, we could definitely work on it.
It that is the case, could you open a GitHub issue <https://github.com/Nixtla/statsforecast/issues/new?assignees=&amp;labels=enhancement&amp;template=feature-request.yml&amp;title=%5B%3CStatsForecast+component%3A+Models%7CCore%7Cetc...%3E%5D+|here>.

Hi <@U035DR8HD6D>, Thank you for your reply. There is already a ticket for this request : <https://github.com/Nixtla/statsforecast/issues/382>

Hi <@U035DR8HD6D>, in the mean time is there any other way I can get the fitted value while I am using spark dataframes (distributed environment)

hey <@U04UECRAUN7>! An alternative would be to modify by hand the `FugueBanckend`  class to return the fitted values (<https://github.com/Nixtla/statsforecast/blob/main/statsforecast/distributed/fugue.py#L47>).

and end users should not need to use FugueBackend, fugue should be implicit now

hey <@U04GA0TR1HU>! This is really cool; we haven’t tested the forward method using a single point; let me check what would be happening and get back to you :slightly_smiling_face:

Hi! Thanks for the question. You might find the cross validation functionality useful: <https://nixtla.github.io/mlforecast/docs/end_to_end_walkthrough.html#cross-validation>

Hi Max, what changes are suppose to be made in the CV function for it to predict on train data

Hi Darius! That is currently not possible.

I'm interested in learning why you would like to use prophet?

Hello <@U035DR8HD6D>!

I am currently working on a demand forecasting project, in which we are using multiple models for generating the predictions. Prophet is one of them and we've seen so far that it also has good accuracy on our datasets. We are also using WindowAverage from your library and  we thought maybe we could wrap the AutoARIMAProphet and WindowAverage models in a single StatsForecast object such that when we generate forecasts using `.forecast()`  the output will be for both models.

Great! We have seen great improvements in using other models for demand data. Maybe you can try this End to End Walkthrough. The idea is to train different models for all unique combinations and then select the best model based on the past perfomance.
For Demand data we have seen great results using; MSTL, ETS, CES and CROSTON.

<https://nixtla.github.io/statsforecast/examples/getting_started_complete.html>

hey <@U04CLTA2EJD>! Yes, that’s possible. You have to “register” the external regressor as in prophet

```model = AutoARIMAProphet()
# "register" the external regressor
model.add_regressor('my_external_regressor')

# train df should have the external regressor
train_df = Y_df[['ds', 'y', 'my_external_regressor']]

# train the model
model.fit(train_df)

# create future dataframe
future_df = model.make_future_dataframe(7, freq='M')

# future_df only contains the ds column, you have to add 
# the external regressor
future_df['my_external_regressor'] = ...

model.predict(future_df)```

Hello Guys, I need to do some hyper parameter tunning with MLforecast the data I am using is very large and it is becoming computionally expensive can. you please suggest some resources for light weight hyper parameter tunning.

Hey <@U050PBC2Q73>! Thank you for your kind words and using our libraries. Yes! We are planning to extend the ecosystem to include classification, clustering and other interesting use cases. Please help us opening an issue in our repo, we could start planning how to colaborate there :)

Hi <@U0512USU2KF>! Thanks for the interest. We are happy to jump into a brief call with you if that works. Either way <@U0316L4HWQ7> will provide some examples. 

Sure max we can come up with some time :slightly_smiling_face:

hey <@U0512USU2KF>!

1. That’s perfectly possible with our mlforecast and statsforecast libraries. Regarding statsforecast here’s an example using databricks (<https://notebooks.databricks.com/notebooks/RCG/intermittent-forecasting/index.html?&amp;_ga=2.52200938.2108250371.1680567893-82c188cb-e021-40d7-bdec-55707a9664cd#intermittent-forecasting_1.html|https://notebooks.databricks.com/notebooks/RCG/intermittent-forecasting/index.html?&amp;_ga=2[…]08250371.1680567893-82c188cb-e021-40d7-bdec-55707a9664cd>). The API changed a little bit, now it is not necessary to declare a backend, it is as simple as passing a spark dataframe to the forecast method. MLForecast works similarly, you have to import the `DistributedMLForecast`  class and pass a spark dataframe to the `fit`  method, here’s an example: <https://nixtla.github.io/mlforecast/distributed.forecast.html>.
2. Our neuralforecast library is compatible with GPUs and multinode GPUs, but we are still working on making it compatible with spark.  We haven’t tested it yet, but mlforecast also could work in such environments (through lightgbm). Unfortunately, StatsForecast currently does not support gpu.
3. Bringing your own models is perfectly possible using statsforecast (univariate models). We can help you with that if you are interested. MLForecast can also support custom models but in a distributed environment such as spark it could not be easy since the model needs to be distributed as well (for example, we use synapse ml to train lightgbm). 
4. You can use your own (spark,dask, or ray) dataframe without a problem. The only requirement is to have at least three columns: unique_id, identifying the time series, `ds` , identifying the temporal column, and `y` , the target column. Here’s a description of the input dataset: <https://nixtla.github.io/neuralforecast/examples/data_format.html>.
Let us know how we can help you. :)

We want to bring deepvar as a model in neural forcast

Hi! Thanks for your question <@U0517FJ5K54>.

Did you try setting the `nonnegative` parameter to true?

Here is an example

Hi Max! Thank you very much for your response!

I worked on the method=*‘mint_cov’* instead of the *‘ols’*.

It seems that when using AutoETS and AutoARIMA the *‘mint_cov’* method works well, while when using ETS and ARIMA I receive the error I mentioned. Is this an expected behavior?

hey <@U0517FJ5K54>! Thank you for using hierarchicalforecast. It seems like the issue you’re experiencing is unexpected. xD

Typically, the `mint_cov` approach breaks when the model’s residuals have low variance, such as when the residuals are close to a constant. Could you check if that’s the case? (You can check this by computing the difference between the actual values and the fitted values of the models and after that computing the variance for each model)

The `mint_cov` function computes the variance-covariance matrix of the residuals, which is a challenging problem. This can result in an ill-conditioned matrix that causes the process to break. To address this, you can use alternative approaches like `wls_var` and `mint_shrink`. These approaches can help to reduce the complexity of the variance-covariance matrix and make it more stable. Hopefully, trying these alternative approaches will help to resolve the issue you’re experiencing.

hey <@U04RVB0EJRJ>! Yes, it is possible to use multiple CPUs/GPUs for cross-validation even if you only have a single time series. To do this, you can pass `cpus=1` and `gpus=1` to the Auto classes during hyperparameter optimization. This will allow you to perform parallel processing by indicating the number of resources allocated to each trial. Note that this approach is particularly useful when training models such as NBEATS and NHITS, which are trained using different windows of different series. However, in your case, you will be training them using different windows of the same series.

But in that case using just one gpu should work well

Ok and with the non auto models it is not possible? Or would I just pass a default config and set num_samples=1?

Also is it possible to log each run after each epoch to wandb?

Oh &amp; if I pass a custom config to any of the auto models, I get the following error...

ValueError: Trial returned a result which did not include the specified metric(s) `loss` that `tune.TuneConfig()` expects. Make sure your calls to `tune.report()` include the metric, or set the TUNE_DISABLE_STRICT_METRIC_CHECKING environment variable to 1. Result: {'trial_id': '6f0eddce', 'experiment_id': 'c928f65feeb747c7abbda5181ebdfcd6', 'date': '2023-04-05_13-47-44', 'timestamp': 1680702464, 'pid': 79944, 'hostname': 'shipgpu001', 'node_ip': '10.99.151.1', 'done': True, 'config/h': 7, 'config/encoder_hidden_size': 100, 'config/encoder_n_layers': 1, 'config/context_size': 50, 'config/decoder_hidden_size': 256, 'config/learning_rate': 0.04436648573316852, 'config/max_steps': 10, 'config/batch_size': 32, 'config/loss': MQLoss(), 'config/random_seed': 13, 'config/valid_loss': MQLoss()}

hey <@U0423GYCJSY>! This problem is usually complex if you have a lot of external regressors. One approach would be to iterate over the different combinations of external regressors using cross-validation to recover the features that minimize the validation loss. The set of possible combinations could be created based on domain knowledge to avoid iterating over all of them.

<@U0423GYCJSY> you can try mutual information <https://valeman.medium.com/mutual-information-for-time-series-forecasting-in-python-377d36ded102|https://valeman.medium.com/mutual-information-for-time-series-forecasting-in-python-377d36ded102>

hey <@U04UKKD2F4K>! MLForecast has proven excellent results in sparse-and-short-horizon tasks, with medium to large datasets (for example, sales). <https://www.kaggle.com/code/lemuz90/m5-mlforecast-eval|Here>, you can find an example using the M5 competition (Walmart sales).

Hi All, I recently came across neuralforecast and would like to implement it in my usecase, where I have to predict next basket size of a customer, so I have prepared the dataset at batch level for each customer. And, each batch is having day wise purchases of a customer, recency, interval, frequency, avgsize of his basket, etc. for 5 days window and based on this sequential purchases, I need to predict his next purchase which is my y_batch..Can anyone please help me how can I use neuralforecast as I have multiple features i.e. 13 features in my batch. Please let me know in case of any questions.
Thanks in advance!

Hi, I wonder if mlforecast supports categorical static features?

Screenshot 2023-04-15 at 12.18.06 AM.png

Screenshot 2023-04-15 at 12.17.57 AM.png

hey <@U053D3591PC>! Thanks for using mlforecast.I saw that you’re passing a set to the `target_transforms`  argument. Could you try using a list? (`target_transforms=[Differences([1, 3, 6, 9])]`  instead of `target_transforms={Differences([1, 3, 6, 9])}` )

Thank you! This solves my problem. Also, I wonder if mlforecast supports feature importances when using xgboost and random forest?

hey <@U053D3591PC>! cool.

Yes, that’s possible. Once the models are fitted, you can access them through the `model_`  attribute. And then you can retrieve the feature importance of the features using something like the following:

hey <@U0423GYCJSY>! Regarding the first warnings, we are working on them, you can follow the discussion on <https://github.com/Nixtla/statsforecast/issues/467|this issue>.

Regarding the ‘tiny datasets’ error, it arises if the series is too short (usually less than 4 or 5 observations). In those cases, we recommend using the `fallback_model`  argument of the `StatsForecast`  class to prevent the pipeline from crashing.  That model will be used when an error appears. Here’s an example:

```StatsForecast(..., fallback_model=Naive())```

hey <@U04UFRXRVAT>! <https://nixtla.github.io/mlforecast/distributed.forecast.html|Here> you can find the documentation of the `DistributedMLForecast`  class which is compatible with spark, dask, and ray.

In particular, <https://nixtla.github.io/mlforecast/distributed.forecast.html#dask|here> you can find an example using dask.

Regarding the error using pyspark, could you provide some details? A screenshot of the error and a code snippet might work

Hi fede I do not get errors the code just crashes I am basically converting a very large dataset to to pyspark dataframe and then applying a pandas udf on this dataframe. My pandas udf has MLforecast running. When i do the above mentioned operation the code crashes

hey <@U054TEFN9U4>! Yes, neuralforecast requires pytorch 2.0 since the newest version of ligthning only supports that version. An option would be to fix the neuralforecast version, including torch: `neuralforecast&lt;1.5.0 torch&lt;2.0.0`

Google is inheriting Nixtla’s notation 

We are doing something right!!

Thanks <@U055A6ADALT>! We will come back to you soon :)

<@U055A6ADALT> you can check <https://github.com/valeman/awesome-conformal-prediction|https://github.com/valeman/awesome-conformal-prediction> and also my forthcoming book on conformal prediction <https://github.com/valeman/awesome-conformal-prediction|https://github.com/valeman/awesome-conformal-prediction> and follow me on LI and Twitter (links on repo)

Thanks David! If you want the master of conformal prediction himself, you should contact <@U03L39VLJBA>. He has been a great support and mentor on conformal. 

<@U055NUK24N5> you check <https://github.com/valeman/awesome-conformal-prediction|https://github.com/valeman/awesome-conformal-prediction> and also my forthcoming book on conformal prediction <https://github.com/valeman/awesome-conformal-prediction|https://github.com/valeman/awesome-conformal-prediction> and follow me on LI and Twitter (links on repo)