Hi All - any recommendations for what objective metrics would be good for very high seasonal spike data? i have a modeling issue where i have a time series which is consistent but low volume all year.. then during a later part of the year, there are a few VERY high weeks of orders. . like.. 10-20x the volume of off season. We have features in the model describing the 'size' of the spike.. based on last years spikes.. but using Tweedie=1.5 it is not capturing the spikes well at all. Lowering tweedie variance param helps a little, but not sure if there is a better metric than tweedie. Thoughts?

my thoughts here (not claiming it is all correct): the question is if one model can do it all. sometimes you need a model to forecast baseline demand and a second to model the crazy peaks. if you want to use one model that should get the crazy peaks right, you might want to stick to mse (maybe even use a weighted loss function, which you can easily integrate into xgb using nixtla) as evaluation function and maybe even mse as objective functon. I would try mse and tweedie as objective during tuning and then evaluate on mse. but in the end you have to test and see what works best, of course. it is always hard to judge without looking at the data. on top of that: do you have other exogenous feature, how do you encode categorical features (sometimes target or catbooost works better than OHE). maybe this gives you some inspiration on what to try next 🙂 p.s.: you could maybe do the following: build a baseline model and a second model (a residual model) to forecast the peaks, which takes the residuals of your fc and the true peak and then corrects the initial forecast to match the peaks better

Appreciate the reply Jan. We thought about that separate model idea and do the base + lift concept. It might still be the direction we have to go, but it doesnt fit as well with all our model features so was hoping to avoid that. I will explore MSE - and see how that might help. Thank you!