https://github.com/nixtla logo
j

J.

09/07/2023, 2:07 PM
Regarding my initial question (https://nixtlacommunity.slack.com/archives/C031EJJMH46/p1693989589210909) I would like to ask if my approach is the recommended way to do an extensive cross_validation in the special case that the forecasting horizon is only 1? I am asking because all examples in the documentation show the usage of 3<n_windows<20 which essentially means only "20 crossvalidated days " in the case of h=1. Also, I would like to ask if the Auto* Models perform hyperparameter tuning and model selection for each window inside the cross validation? Or, do they only perform one initial model selection on the whole time series, find the best model, and use that best model for each window inside the cross validation? Thank you in advance!
j

José Morales

09/07/2023, 4:51 PM
Hey. With respect to the Auto models, they find the best parameters for each window
j

J.

09/17/2023, 1:48 PM
@José Morales Do you know if the same is true for the neuralforecast Models? Judging be the time needed to finish the AutoMLP cross-validation (which takes roughly 30secs on my 4090 and my dataset, and about 1,9 hours for AutoARIMA on the same dataset) it seems like neuralforecast does one hyperparameter search per series and not per window. Is that correct?
j

José Morales

09/18/2023, 4:32 PM
Hey. neuralforecast uses a single model for all series, so it's still per window but it only searches n_windows instead of n_windows * n_series
j

J.

09/20/2023, 10:20 AM
@José Morales Hey, so it randomly chooses one series X from my "big dataframe" and then for each window in X it chooses the best parameters? So it has one parameter combination for one particular window and uses that parameter combination for all the corresponding windows from the N-1 time-series? Wouldn't that mean that the hyperparameter tuning highly depends on the chosen time-series X and not on all time-series?
Or do you mean that it uses a single model which is trained on all time series (global model)?
j

José Morales

09/20/2023, 2:58 PM
It uses a global model
🙌 1