Regarding my initial question (https://nixtlacommunity.slack.com/archives/C031EJJMH46/p1693989589210909) I would like to ask if my approach is the recommended way to do an extensive cross_validation in the special case that the forecasting horizon is only 1? I am asking because all examples in the documentation show the usage of 3<n_windows<20 which essentially means only "20 crossvalidated days " in the case of h=1. Also, I would like to ask if the Auto* Models perform hyperparameter tuning and model selection for each window inside the cross validation? Or, do they only perform one initial model selection on the whole time series, find the best model, and use that best model for each window inside the cross validation? Thank you in advance!

Hey. With respect to the Auto models, they find the best parameters for each window

@José Morales Do you know if the same is true for the neuralforecast Models? Judging be the time needed to finish the AutoMLP cross-validation (which takes roughly 30secs on my 4090 and my dataset, and about 1,9 hours for AutoARIMA on the same dataset) it seems like neuralforecast does one hyperparameter search per series and not per window. Is that correct?

Hey. neuralforecast uses a single model for all series, so it's still per window but it only searches n_windows instead of n_windows * n_series

@José Morales Hey, so it randomly chooses one series X from my "big dataframe" and then for each window in X it chooses the best parameters? So it has one parameter combination for one particular window and uses that parameter combination for all the corresponding windows from the N-1 time-series? Wouldn't that mean that the hyperparameter tuning highly depends on the chosen time-series X and not on all time-series?

It uses a global model

@José Morales When using neuralforecast, I changed the n_windows parameters from 1 to 700 and it did not have any negative impact on the cross-validation speed. On the contrary, it was even faster when using a bigger number for n_windows (700 instead of 1)? I took roughly 350 seconds using n_windows=1 and roughly 220 seconds when using n_windows=700. Are you sure that there is hyperparameter search per window? For me, it seems like it is searching parameters on the remaining data (data that does not interfere the first day to be forecasted in the cross-validation data) for tuning the best hyperparameters and then uses this model for cross-validation. Or why is the process faster when doing more n_windows?

Can you share the code you're running?

Hi @J. ! The hyperparameter is done once for the entire dataset since the model is global. The

n_windows

only controls the length of the validation set. Each model with a particular configuration is trained once on the train data, and the validation loss is computed once for the entire validation set (consisting of

n_windows

). A longer

n_windows

only increases the length of the validation set. It does not increase the total time because the inference on the validation set is done efficiently on batches.