Very nice prediction intervals with conformal prediction magic and Nixtla. Kudos to the team @Max (Nixtla) @Kevin Kho @fede (nixtla) (they/them)

have the intervals been benchmarked on M4?

@Tyler Blume the examples are from M4, benchmarked against what? Most of the probabilistic submissions for M4 were invalid prediction intervals there is a paper about it.

yeah but how are the conformal intervals from StatsForecast doing on it? Just curious if it's been tested yet.

Hi @Tyler Blume, due to temporal constraints (no pun intended) we haven't run benchmarks on that dataset against other 'classical' ways of quantifying uncertainty. If you feel like taking shot I'm sure @Valeriy and the community would love that. My money is on conformal prediction, though.

@Tyler Blume this is a good point. As I understand it is still in development and has not been released yet. Same as @Max (Nixtla) my money is on CP as well just because multiple other libraries implemented it and saw great results. In fact just yesterday I had someone also implementing it privately over the weekend for large financial corp to forecast volatility using proprietary models and also saw great results straight away. The tech is rock solid there are different ways to implement it and there is where experiments and benchmarking should figure out the best variant.

Thank you @Valeriy! 🙌

started messing around with the conformal intervals for AutoETS getting the percentage that fall in each interval for a few different window sizes, no idea how to actually grade the performance though @Valeriy any recommendations or is the percentage in the interval vs desired percentage a decent measure? Prediction interval stuff is a new world for me 🙂

@Tyler Blume you can use the following: 1) coverage vis-a-vis used confidence level 2) efficiency aka width of intervals the two covered here https://medium.com/@valeman/how-to-evaluate-probabilistic-forecasts-ace8b7ad3491 also recommend to use CRPS. https://medium.com/trusted-data-science-haleon/metrics-for-distributional-forecasts-60e156c60177