An LSTM model that is trained on hourly data from 01-01-2018 to 31-12-2022 and is trained to forecast the next 24 hours (horizon = 24). Why is it only possible to forecast the full day for 01-01-2023 and not the full day for 02-01-2023 and how can I change the model to allow for this? I get this error: ValueError: There are missing combinations of ids and times in

futr_df

.

Hello! Does

futr_df

has the timestamps of the full day of 02-01-2023 and does the

unique_id

column matches the one used in training?

Yes, Image1: test dataframe (future df) Image2: train dataframe

So it works for 01-01-2023, right?

Yes!

My understanding is that you're doing some kind of manual cross-validation. You can use the cross-validation method and specify the validation (if any) and test size instead, using a horizon of 24h. Otherwise, you need to manually update the input df with new values and then forecast the next 24h. Right now, you model only trained until the end of 2022. So when you ask for prediction on Jan 2. 2023, it's like asking for an horizon of 48 when the model was trained to predict 24h. You can check this tutorial on cross-validation. I hope this helps!

The idea is to do the following @Marco (see image attached: visualization of a recursive multi-step multi-output model), so you train on full history of 2018 to 2022 and then start doing inference. At inference time you have a 7 * 24H input_size (168 lagged prices, but also other covariates) and in every iteration you do a forecast for next day, then the model steps 24 timesteps ahead and does another forecast. We wanted to only retrain the model after 7 days of forecasting, so recalibrate (retrain with new data) after 7 days. Is this possible with the cross-validation method? Or what should I use for that?

By default, the LSTM model is autoregressive, just like the picture you're showing. So, by setting an horizon of 7 for example, it automatically does one prediction at a time and uses that prediction to generate the next, until the entire horizon is covered. So in your case, just set a longer horizon! Cross-validation allows you to test multiple forecast windows. This is a way to ensure your model is robust and not just good by chance. You can look here for an illustration of cross-validation. I hope this helps!

@Marco Setting a forecast horizon of 365 * 24 requires a 240GB CUDA memory allocation? This is a crazy amount? How do you deal with this?

You can try reducing

batch_size

and

valid_batch_size

.

Hello everyone, I'm here on this open topic because I have the same error of AK. As him, I have a 2018-22 dataset for training and validation, and 2023 for test. I want to check the testing error, but I got the same errors (You can run the

make_future_dataframe()

method to get the expected combinations or the

get_missing_future(futr_df)

method to get the missing combinations.) only in the multivariate case. In the univariate one, I can do a simply loop to predict the whole year, then calculating the error (it seems working):

horizon = 24

past_days = 1

y_test = dataset_input.loc[dataset_input['Time_data'] >= '2023-01-01']

y_test = y_test.query('unique_id in @uids').reset_index(drop=True)

daily_test= y_test.groupby(y_test['Time_data'].dt.date)

y_test_list= []

y_pred_list = []

for date, day in daily_test:

y_test_daily = day.copy()

y_test_list.append(y_test_daily)

for i in range(past_days*horizon, len(y_test), horizon):

y_test_block = y_test.iloc[i-past_days*horizon:i]

y_pred = nf.predict(y_test_block)

y_pred_list.append(y_pred)

But in the multivariate case, (i report only a snip of code)

daily_test= y_test.groupby(y_test['Time_data'].dt.date)

y_test_list= []

y_pred_list = []

futr_exog_list = ['Ora', 'GiornoSettimana', 'holiday', 'SOLAR', 'WIND']

hist_exog_list = ['LOAD', 'GAS', 'PAST_PRICE']

for date, day in daily_test:

y_test_daily = day.copy()

y_test_list.append(y_test_daily)

y_pred = nf_multi.predict(futr_df = y_test_list[0][['unique_id', 'Time_data'] + futr_exog_list])

y_pred_list.append(y_pred)

y_pred = y_pred.reset_index()

I got the same error (ValueError: There are missing combinations of ids and times in

futr_df

. You can run the

make_future_dataframe()

method to get the expected combinations or the

get_missing_future(futr_df)

method to get the missing combinations), and I'm able only to predict the first day of 2023. I wanted to use cross validation, but I have some trouble to deeply understand how it works, so I decided to use the simpler fit/predic methods. Let me know if it's clear. (I have also a simper question about exogenous variables. On the site it's reported that "_There is no need to add the target variable

y

and historic variables as they won’t be used by the model."_ So the model will only use that features, ignoring his past? Also, can i use crossvalidation with multivariate dataset?)

Hello! You shouldn't have to do any loops to forecast using the libraries. You can either run cross_validation and specify the test and validation sizes, as well as the step size, or specify a number of cross_validation windows. The package then handles the rest. You can see our detailed tutorial on cross-validation for more information.

@Marco but can I use also exogenous variables? (if yes, is it true that the model won't use historic and target variables?)

Yes, you can use exogenous variables if the model supports them. For example, TSMixer and NHITS both support exogenous features (there are of course more models that support them)

but then how can i make a final evaluation on a test set? What I've understood is: I have a dataset covering 2018-2024 (till now). I want to use 2024 data for test set, and the rest for training and validation, using cross_validation method. What is the procedure in this case? (it's a pseudo code)

horizon = 24 # one day forecast

past_days = 1

A.

y_df = the whole dataset (including 6 month of 2024)

nf_preds = nf.cross_validation(df=y_df, val_size = 24*345, test_size = 24*365/2 (6months), id_col='unique_id', verbose=True, refit=False)

B.

y_df = dataset until 12-31-2023

nf_preds = nf.cross_validation(df=y_df, step_size = horizon, n_windows=365, id_col='unique_id', verbose=True, refit=False)

and then test the model using nf.predict on the 2024 dataset (and here the question: how can I do a test on 6 months without looping) I've obviusly read your tutorial, but still I don't properly get how it's done. Thanks for your pacience!!!

It would option A in your case.