https://github.com/nixtla logo
v

Valeriy

08/22/2023, 2:08 PM
Very nice prediction intervals with conformal prediction magic and Nixtla. Kudos to the team @Max (Nixtla) @Kevin Kho @fede (nixtla) (they/them)
❤️ 1
t

Tyler Blume

08/22/2023, 2:14 PM
have the intervals been benchmarked on M4?
v

Valeriy

08/22/2023, 4:11 PM
@Tyler Blume the examples are from M4, benchmarked against what? Most of the probabilistic submissions for M4 were invalid prediction intervals there is a paper about it.
t

Tyler Blume

08/22/2023, 4:19 PM
yeah but how are the conformal intervals from StatsForecast doing on it? Just curious if it's been tested yet.
👍 1
m

Max (Nixtla)

08/22/2023, 7:11 PM
Hi @Tyler Blume, due to temporal constraints (no pun intended) we haven't run benchmarks on that dataset against other 'classical' ways of quantifying uncertainty. If you feel like taking shot I'm sure @Valeriy and the community would love that. My money is on conformal prediction, though.
👍 2
v

Valeriy

08/22/2023, 7:26 PM
@Tyler Blume this is a good point. As I understand it is still in development and has not been released yet. Same as @Max (Nixtla) my money is on CP as well just because multiple other libraries implemented it and saw great results. In fact just yesterday I had someone also implementing it privately over the weekend for large financial corp to forecast volatility using proprietary models and also saw great results straight away. The tech is rock solid there are different ways to implement it and there is where experiments and benchmarking should figure out the best variant.
f

fede (nixtla) (they/them)

08/22/2023, 7:43 PM
Thank you @Valeriy! 🙌
👍 1
t

Tyler Blume

09/13/2023, 2:40 PM
started messing around with the conformal intervals for AutoETS getting the percentage that fall in each interval for a few different window sizes, no idea how to actually grade the performance though @Valeriy any recommendations or is the percentage in the interval vs desired percentage a decent measure? Prediction interval stuff is a new world for me 🙂
👍 1
v

Valeriy

09/13/2023, 5:32 PM
@Tyler Blume you can use the following: 1) coverage vis-a-vis used confidence level 2) efficiency aka width of intervals the two covered here https://medium.com/@valeman/how-to-evaluate-probabilistic-forecasts-ace8b7ad3491 also recommend to use CRPS. https://medium.com/trusted-data-science-haleon/metrics-for-distributional-forecasts-60e156c60177
👍 2