Hi Nixtla Team, I get very good results with the cross validation method, but I get not even comparable way worse results using the forecast method with time-gpt-1-long-horizon model even when I try to forecast the same time periods I forecasted with the cross validation. Is here a reason for? I am using exactly the same data setup for both methods, and in my understanding the results should be somewhat similar as it trains on the same previous data when trying to forecast the time periods and with the same model setup with the same finetunesteps and finetune loss function etc. So am I missing something? Like the difference is consistently way worse for foracast which does a very crappy forecast repeating the same "seasonality" forecast over and over for x number of days. And with cross validation it actually predicts a very good price prediction in the future which is not a repeated forecast of the same over and over again.

To make sure I understand, are you for example doing cross-validation over 36 time steps with 3 windows of 12 predictions vs predicting the entire 36 time steps in one shot?

no if fore example the cross validation is 3 windows of 12 predictions then I do the forecast methoed also 12 time steps repeated in each of the three windows

Thanks @Jelte Bottema! We're taking a look at what you shared in email and will get back to you here.

Alright, so we made some tests on our side, and the only explanation is that you have other exogenous features in your data. Basically, during cross-validation, the model accesses the future values of exogenous features to inform the predictions. When using the predict method, if you don't pass the future values of exogenous features in

X_df

, then the model doesn't access them, which changes the predictions significantly.

Thanks for the response, this was indeed the case but regarding this two questions: Normally if I would predict X steps in the future I will not have access to the future values of the exogenous variables for the X steps in the future. That is why I want them to use as historical variables in order to predict X steps in the future such that I could make a prediction right now in time X steps in the future: 1. Why does the cross validation method uses the future values of the exogenous variables and is there an option where it does not use the future values of these exogenous variables 2. Or is my thinking of future exogenous variables wrong and do you input them as well in the forecast method(without using the prediction of your exogenous variables method with time-GPT model but only using the data at hand)

At the moment no, cross-validation will automatically take into account future values of exogenous features. Therefore, you need to filter them out before running cross-validation if you don't want to use them By using historical features only, the model tries to make predictions using the known values in the past, which can sometimes make the predictions worse than simply ignoring them. Here, it seems to be the case. You can try to forecast the exogenous features and use those predictions to make predictions. We have a detailed tutorial on that here.