Hello, happy Monday! A question from a self-declar...
# supportt
Tracy Teal
07/15/2024, 4:44 PMHello, happy Monday! A question from a self-declared fan of Nixtla, Asad Shoaib from WeCrunch, on hyperparameter tuning, likely for @Cristian Challu.
✅ 1
Tracy Teal
07/15/2024, 4:45 PMI hope this message finds you well. I am writing to discuss an issue I've encountered while working with Nixtla's platform, specifically regarding hyperparameter tuning and its impact on model output consistency.
Recently, I have been experimenting with Nixtla's DilatedRNN using default configurations, alongside running its auto-hyperparameter tuning function (AutoDilatedRNN) with identical settings over 60 months of data. My objective was to achieve consistent output across both approaches.
However, despite setting the configurations the same for both scenarios, I've observed discrepancies in the error returned by each. Upon closer examination, I've identified that during hyperparameter optimization, the model utilizes a validation set in addition to the training data. This results in the training data being limited to 48 months, with the remaining 12 months allocated to validation.
To align the conditions with the default model, I reduced the training data for the default call to 48 months. Surprisingly, the outputs still do not match, suggesting underlying factors influencing the results beyond the training data duration.
Given these findings, I am reaching out to your team to seek clarification on how one can achieve consistent results when running hyperparameter optimization, ensuring they align with those from the default model setup. Could you please provide guidance or insights into this matter?
I appreciate your prompt attention to this issue and look forward to your valuable input. Please let me know if there are any additional details or data I can provide to facilitate this discussion.
Thank you for your assistance.
Best regards,
Asad Shoaib
Big fan of Nixtla (& AI Product Manager)
WeCrunch
Tracy Teal
07/15/2024, 4:45 PM@Cristian Challu I could reply and say we're happy to discuss and set up a call, rather than try to reply via email.
m
Marco
07/15/2024, 5:25 PMSo, the default configuration in AutoDilatedRNN and the default parameters of DilatedRNN are two different things.
In AutoDilatedRNN, we set a default feature space for tuning the hyperparameters, but the user can define its own space. The result of running the auto model is the best combination of hyperparameters from the defined configuration.
The default values of DilatedRNN are not necessarily optimal, and so likely different than the trained AutoDilatedRNN. They're just default values such that the model can run, but they are not optimized.
So, that's why he's getting different results, In my opinion, he should just run the Auto models and get the best configuration, so the model can be reused later.
But ultimately, we can take any model, and as long as the parameters and datasets are set, the results will be the same. Here, the hyperparameters are different.
t
Tracy Teal
07/17/2024, 8:04 PMThanks @Marco! Shared with Asad.
Tracy Teal
07/18/2024, 11:24 PM@Marco Asad appreciated the response, thank you! And he replied. 🙂 His response here. Any thoughts on a follow up?
We can do something short, and I can direct him to our community Slack if he wants more of a discussion with you.
Thank you for the detailed response. I agree that the default in DilatedRNN and AutoDilatedRNN are two different things - the former is a single set of parameters (which may not be optimal) and the latter, like you defined, is a parameter space containing different sets (and thus may have the optimal set).
I also concur with how as long as parameters and datasets are same, the result will be the same irrespective of the model.
In my case, I am trying to set the same parameters (on the same dataset) for both the models (AutoDilatedRNN and DilatedRNN) but am unable to get the same result. I looked at Nixtla's github's code for NeuralForecast (dilated_rnn.py) for the default values (which again, may not be optimal) of parameters for DilatedRNN and am specifying just that set in hyperparameter space for AutoDilatedRNN. Hence, I am trying to call the same parameters for both the models. However, I am not able to get the same output.
Perhaps I am missing something here and I would really appreciate any feedback that you may have on this. I have attached the pdf of the configuration I am passing for the Auto model to this email FYI.
Also, I am aware that the Auto model takes a validation set (which I set to 12 out of 60 months total). But even when I restrict the default model's training data to 48 months (to match Auto's training data), I am unable to get the same result.
Looking forward to hearing from you soon.
Many thanks in advance once again!