Folks who are experienced with NBEATS, which param...
# neural-forecastf
Farzad E
02/10/2023, 5:21 PMFolks who are experienced with NBEATS, which parameters are the most important ones to tune? I have been playing around with n_harmonics, n_polynomials and learning_rate but it is clear that the model is not learning anything. R2 barely reaches 50% and MAPE is at 100%! I'm using 4 years of daily data to predict 2022. My data doesn't look that difficult to predict. I must be missing something simple. Here is the fit (magenta is the actual and cyan is the NBEATS:
Farzad E
02/10/2023, 5:22 PMThis is my code:
Farzad E
02/10/2023, 5:22 PMAnd the full data from 2018 onward:
Farzad E
02/10/2023, 5:26 PMThe interesting thing is that it never learns the amplitude changes at the daily level. It tries to follow the changes of the amplitude over the year (although poorly) but it's not even close when it comes to the amplitude changes at the daily level. Completely ignores the signal amplitude there.
c
Cristian (Nixtla)
02/10/2023, 5:47 PMHi @Farzad E. This is actually a relatively hard problem given the horizon size of 365. We have a specific paper and model for long horizon forecasting (https://arxiv.org/abs/2201.12886) in our library, the NHITS. Our experiments indeed show that the NBEATS interpretable (the default configuration, what you are running) does not perform well with long horizon.
Cristian (Nixtla)
02/10/2023, 5:49 PMI recommend you to use the
AutoAutoNHITSAutoNBEATSCristian (Nixtla)
02/10/2023, 5:52 PMFor the specific case of the NBEATS, we don' usually change the n_harmonics and n_polynomials. To make the model more flexible you can increase the number of blocks (
n_blocks[3,3,3]mlp_unitsCristian (Nixtla)
02/10/2023, 5:53 PMFinally, adding exogenous calendar variables might help, such as month dummies. Here we have the documentation for adding variables: https://nixtla.github.io/neuralforecast/examples/exogenous_variables.html
Cristian (Nixtla)
02/10/2023, 5:59 PMLooking at the complete data, 2022 is also hard to forecast because it has larger values than previous years, thats probably why the forecasts are lower. I strongly recommend you to try the NHITS model, we designed it specifically for this setting. Hope it helps!
f
Farzad E
02/10/2023, 6:15 PM@Cristian (Nixtla) This is incredible help. Thanks a lot! I didn't know about limitations of NBEATS. I'll look into NHITS and will try the Auto versions as you mentioned.
Farzad E
02/10/2023, 6:24 PM@Cristian (Nixtla) On a related note, I know classical methods like ARIMA have difficulty forecasting series with long periods (e.g., 365 or even 52) and that's why I started looking into deep learning methods. What are your thoughts on aggregating data and forecasting at weekly level? Do most deep learning forecasters have a hard time with weekly data as well like ARIMA has or will they perform much better at weekly compared to daily?
m
Martin Bel
02/21/2023, 7:39 PMweekly data is hard with any method, at least in my experience.
👍 1
f
Farzad E
02/21/2023, 7:45 PM@Martin Bel what is your go to method for weekly/daily? And what type of data do you work with of you don't mind me asking? My current project is forecasting weekly and daily for the housing market data.
m
Martin Bel
02/21/2023, 7:52 PMI'd start with the statsforecast methods first, AutoArima for example is generally hard to beat. Then you can try the mlforecast package, this should be simpler to use than NHiTS.
👍 1
f
Farzad E
02/21/2023, 7:55 PMArima won't work for period of 365 and even for 52 it takes forever to compute. It's more practical for monthly or quarterly forecasts at least in my experience. I haven't tried mlforecast so thanks for mentioning that. I'll look into it.
m
Martin Bel
02/21/2023, 7:57 PMIt works ok for 365 or 52 periods. But I've seen this being slow also. Compared to R much slower in these frequencies.
This doesn't make it a "bad" model for this problems. It's generally the main model.
👍 1
f
Farzad E
02/21/2023, 8:00 PMYeah it does work like it gives you results. What I meant was that it becomes so slow specially if you have a few years of data and hundreds of time series, that in practice it won't work for your project. I haven't coded in R in years but I hear it is a bit faster in forecasting but not sure if that would help much.
m
Martin Bel
02/21/2023, 8:01 PMif you have hundreds of time series, perhaps you can use hierarchicalforecast.
It always depends on the data. But it's a hard to beat benchmark.
👍 1
f
Farzad E
02/21/2023, 8:04 PMThanks. I'll check it out.
2 Views