I'm comparing nhits with and without a particular xreg. I thought of fitting nhits two times 1. Using nhits without the xreg 2. Using nhits with xreg Then compare the rmse. I only use scaler_type = "robust" when I include the xreg. Does that have any effect on the comparison, i.e. do I need to use scaler_type = "robust" in both models to make the comparison as "correct" as possible? What I'm seeing is this: a) model without xreg and without scaler_type = "robust" -> rmse = 10 b) model without xreg and scaler_type = "robust" -> rmse = 15 c) model with xreg and scaler_type = "robust" -> rmse = 12 Should I compare (c) to (b) or (a) to get a sense if the xreg is helpful or not?

I think that if you're inverse transforming your predictions and measuring performance on the original scale of the data, then the comparison is valid. The scaler is just a preprocessing step. Note that by default, the scaler for NHITS is "identity".

Ok thanks :)