Hi, I am wondering what is the proper way to perform in-sample evaluation? For example, the transformer-based model is trained to learn within window behavior, where we are given data

input_size

, but asked to make prediction on a data of

h

. The goal is to make evaluate over randomly chosen window on the validation dataset. Currently, I am thinking of doing resampling myself, but I am wondering if there is in-built features for insample evaluation. I think

predict_insample

might be doing this, but I am confused about what the output of predict_insample is. Does it only contain the predicted value? Is it a fair metrics to simply compute accuracy of predict_insample with the original df as a measurement of method.

Hey @Yang Guo, The insample validation evaluation is a feature that we have not implemented. It would require to modify BaseWindows validation step to sample within.

Could you please expand on that? I thought I could simply use the

nf.predict

and directly apply subset of the dataset.

Hi @Yang Guo, we have other options as well. For example, the

cross_validation

method uses a validation set (of size

val_size

) for model selection, and then automatically produces the forecast for the entire test set (of size

test_size

or

n_windows

of size

h

).

I think the issue is that predict_insample can only apply to the default dataset. So I think you point is that after performing the cross validation, then apply predict_insample?

Can you provide more details on the pipeline you are trying to build? No,

cross_validation

only trains the model once.

Actually, let me rephrase my question. Cross_validation performs actual training on the dataset though with some additional features. To evaluate the model, I cannot use the same dataset used in the cross_validation dataset. However, predict_insample does not take dataset as input.

If your objective is to compare between models on historic data,

cross_validation

is the way to go. This is the function we have used in our published research, and it is the standard way of comparing the performance of models. The historic data is separated chronologically in train/val/test. Models are trained on the train set, and then it uses the validation set for model selection and hyperparameter tuning (for example, if you use an

auto

model such as the

AutoPatchTST

). Finally, it returns the forecast on the test set, which was never seen by the model during training.

I agree with this pipeline, but is there any way that I can simply evaluate a trained model? The evaluation is done in a similar way to predict_insample but on a different dataset.

Ok, I understand you point now. So essentially you want to train a model on one dataset and predict on a new one? (not necessarily with the same time series)

Yep

You should use the

predict

method for that, but as you said, it can only forecast one window. You just need to send

predict(df=new_df)

. We actually have a tutorial on transfer learning with this use case here: https://nixtla.github.io/neuralforecast/examples/transfer_learning.html