This message was deleted.
# statsforecasts
Slackbot
11/08/2023, 8:58 PMThis message was deleted.
j
José Morales
11/08/2023, 9:41 PMHey. Which model are you using?
b
Brian Head
11/08/2023, 9:42 PMHey @José Morales,
I'm currently using:
AutoARIMA, AutoETS, AutoCES, AutoTheta, DynamicOptimizedTheta, Holt, HoltWinters
j
José Morales
11/08/2023, 9:44 PMDo all the models yield the constant prediction?
b
Brian Head
11/08/2023, 9:45 PMI'm 99% sure they do. My script right now is selecting the best model based on MAE. Among the "best models" they seem to all have constant predictions.
Brian Head
11/08/2023, 9:46 PMI had just restarted a run with h=11 before you responded so I'm waiting on it to finish so I can go back and look.
Brian Head
11/08/2023, 10:39 PM@José Morales confirmed by doing h=6, h=11, and h=12. First two I get non-constant predictions. But, for h=12 I get constant predictions for all of the unique_ids. This is a random sample of 5% of my overall unique ids and results for the best model based on MAE, but I wouldn't expect those to change the results. Any ideas?
j
José Morales
11/08/2023, 10:44 PMHow are you computing the MAE? Is it possible that for h=12 the best model is the constant one?
José Morales
11/08/2023, 10:44 PMIt'd be great if you could save the individual forecasts before selecting the best one so that we at least know where they're coming from (which model)
m
Max (Nixtla)
11/09/2023, 4:30 AMHi @Brian Head, this could be an expected behavior. Think of it like this, the further ahead you want to predict the future, the better a 'straight' constant line might satisfy the objective of minimizing a a specific error metric, in this case MAE. In other words, sometimes the best some models can do based on the data provided is a straight line.
In the specific case of ETS, sometimes, without seasonality restrictions, it oddly opts for a straight-line forecast. We've put together a Google Colab tutorial (https://colab.research.google.com/drive/1mKYY4elPDjm4XRjyKxk9ZnWmYZQyBRLE#scrollTo=HRTvP8UdS8Wo) showing how we can guide it better by restricting seasonality. It should make a noticeable difference.
b
Brian Head
11/09/2023, 3:29 PM@José Morales and @Max (Nixtla) thank you both for taking the time to help me out. It is very much appreciated.
@José Morales: I've saved the individual forecasts and this is the case for all models, not just the "best" model based on MAE. Horizon of 3, 6, 9, 11 all get non-constant predictions for most unique_ids (although there were a few in the h=11 run that had the constant prediction issue). However, when I try h=12 or h=18 I get constant predictions for all models across all unique_ids.
@Max (Nixtla) I also tested the seasonal/nonseasonal AutoETS trick you shared. Unfortunately, I get the exact same results for both model types and this is true for all unique_ids (each unique id has it's own predicted values, but all of the predictions for any particular unique_id is the same, when using an h greater than 11). It just seems odd to me that for many unique series I'd get the straight/constant prediction. I've paired down my code to exclude exogenous variables, removed the future frame entirely, and tried on multiple series. Same results.
Any other thoughts/ideas would be much appreciated.
Brian Head
11/09/2023, 3:30 PMintervals = ConformalIntervals(h=toggle, n_windows=5)SF_models = [# AutoARIMA(prediction_intervals=intervals, alias="ARIMA"), #arima works for in sampleAutoETS(prediction_intervals=intervals, season_length=12, alias="NonSeasonalAutoETS"), #ets works for in sampleAutoETS(prediction_intervals=intervals, season_length=12, model='ZZA', alias="SeasonalAutoETS"), #ets works for in sample# AutoCES(prediction_intervals=intervals), #CES works for in sample# AutoTheta(prediction_intervals=intervals, alias="Theta"), #Theta1 works for in sample; decomposition_type='additive',# DynamicOptimizedTheta(prediction_intervals=intervals, season_length=13),# Holt(prediction_intervals=intervals, season_length=13),# HoltWinters(prediction_intervals=intervals, season_length=13)]j
José Morales
11/09/2023, 5:40 PMSo the training set is the same and the only thing that changes is the forecast horizon?
b
Brian Head
11/09/2023, 5:43 PMCorrect
j
José Morales
11/09/2023, 6:07 PMIs it possible for you to provide a sample serie that reproduces the issue?
b
Brian Head
11/09/2023, 6:12 PMI wish I could, but not allowed to share any data for security reasons.
j
José Morales
11/09/2023, 6:26 PMYeah I mean if you're able to reproduce the issue with some simulated data
José Morales
11/09/2023, 6:31 PMI'm trying this but it works fine:
Copy code
from statsforecast import StatsForecast
from statsforecast.models import AutoETS
from statsforecast.utils import ConformalIntervals, generate_series
series = generate_series(1, freq='M')
intervals = ConformalIntervals(h=12, n_windows=5)
sf = StatsForecast(
models=[
AutoETS(prediction_intervals=intervals, season_length=12, alias="NonSeasonalAutoETS")
],
freq='M',
)
sf.forecast(df=series, h=12)b
Brian Head
11/09/2023, 6:32 PMLet me see what I can do to create a synthentic/simulated version.
j
José Morales
11/09/2023, 6:32 PMAlso, which method are you using? forecast, fit+predict, etc?
b
Brian Head
11/09/2023, 6:33 PMforecast
6 Views