Hi, how can we do cross-validation to select the b...
# hierarchicalforecast
b
Hi, how can we do cross-validation to select the best model while also using conformal intervals, specifically in a hierarchical context?
o
You have to do the cross-validation per model/hierarchy, and do the hierarchical reconciliation afterwards as a post-processing step.
b
Yes I have that, but how do I add conformal prediction intervals to it as well
o
Maybe I don't understand the question. You can create forecasts, add levels to them using conformal prediction, reconcile them using hierarchical forecast. What specific part of this workflow doesn't work for you and could you please share a piece of code so I can understand where you're experiencing issues.
If it's a question of whether we support conformal intervals in hierarchicalforecast, then the answer is no, you can use one the available intervals methods
b
I wish to create prediction intervals for MLforecast, however the data has a hierarchical structure in which I perform different reconciliation methods. What other interval method could I then use?
I do not have a code yet but I was wondering to what extent I can perform conformal intervals while also doing cross validation in a hierarchical context, as conformal intervals are also based on some kind of cross validation
The code is implemented during training, during training I perform cross validation and then reconcile. Is it possible to include this function in cross validation?
o
No, reconciliation is a post-processing step, outside the training procedure. So for now if you'd want to perform cross-validation on that you'd have to write your own loop. I don't think you can reasonably mix it in with training though, if you want that I'd suggest to use NeuralForecast's HINT model
b
My question is: How can we implement conformal prediction with MLForecast while performing cross-validation to select the best model per unique_id? Since prediction intervals are applied during fit(), should conformal prediction be done separately during the prediction phase in my case? https://nixtlaverse.nixtla.io/hierarchicalforecast/examples/mlframeworksexample.html -> in this link for example, MLforecast and NNforecast are used for probabilistic forecasting in a hierarchical context, the prediction interval method (conformal prediction) is used during fitting
o
You can supply prediction intervals to cross-validation too