https://www.nixtla.io logo
Join Slack
Powered by
Hello, is there any way to save automlforecast obj...
# mlforecast
s
Hello, is there any way to save automlforecast object as we are running the optuna study? Perhaps like this? I tried doing this but it wont save anything
Copy code
def save_best_callback(study: optuna.study.Study, trial: optuna.trial.FrozenTrial):
    """
    Callback to save the AutoMLForecast object when a new best trial is found.
    """
    # Check if the current trial is the best one so far
    if study.best_trial.number == trial.number:
        print(f"\n:tada: New best trial found; Saving it now! Trial #{trial.number}")
        # The 'auto_mlf' object is accessible from the parent scope
        auto_mlf.save('AutoLightGBM')

horizon = 31
my_lgb = AutoModel(
    model=lgb.LGBMRegressor(),
    config=my_lgb_config,
)
auto_mlf = AutoMLForecast(
    models={'lgb':my_lgb},
    freq='1D',
    season_length=7,
    init_config=my_init_config,
    fit_config = my_fit_config,
    num_threads=3
)
auto_mlf.fit(
    processed_df,
    n_windows=12,
    h=horizon,
    num_samples=1000,  # number of trials to run
    loss=custom_loss,
    optimize_kwargs={'callbacks': [save_best_callback]}
)
m
Hi Sarim! We are just testing this new bot. Please let us know if its helpful. @Ask Inkeep, please answer: Hello, is there any way to save automlforecast object as we are running the optuna study? Perhaps like this? I tried doing this but it wont save anything
Copy code
def save_best_callback(study: optuna.study.Study, trial: optuna.trial.FrozenTrial):
    """
    Callback to save the AutoMLForecast object when a new best trial is found.
    """
    # Check if the current trial is the best one so far
    if study.best_trial.number == trial.number:
        print(f"\n:tada: New best trial found; Saving it now! Trial #{trial.number}")
        # The 'auto_mlf' object is accessible from the parent scope
        auto_mlf.save('AutoLightGBM')

horizon = 31
my_lgb = AutoModel(
    model=lgb.LGBMRegressor(),
    config=my_lgb_config,
)
auto_mlf = AutoMLForecast(
    models={'lgb':my_lgb},
    freq='1D',
    season_length=7,
    init_config=my_init_config,
    fit_config = my_fit_config,
    num_threads=3
)
auto_mlf.fit(
    processed_df,
    n_windows=12,
    h=horizon,
    num_samples=1000,  # number of trials to run
    loss=custom_loss,
    optimize_kwargs={'callbacks': [save_best_callback]}
)
a
Thanks for tagging me @Max (Nixtla). I'll look through relevant sources and get back to you shortly.
Looking at your code, you're using the ​*MLForecast*​ library's 
AutoMLForecast
class (1) . The issue with your callback approach is that during the optimization process, the 
AutoMLForecast
object hasn't completed its training yet, so calling 
save()
within the callback won't save a fully trained model . The 
AutoMLForecast
class has a 
save
method that works after the optimization is complete :
Copy code
AutoMLForecast.save (path:Union[str,pathlib.Path])
The proper workflow is to save the 
AutoMLForecast
object after the 
fit
method completes :
Copy code
auto_mlf.fit(
    processed_df,
    n_windows=12,
    h=horizon,
    num_samples=1000,
    loss=custom_loss,
    optimize_kwargs={'callbacks': [save_best_callback]}
)
# Save after optimization is complete
auto_mlf.save('path/to/save/directory')
The 
AutoMLForecast.fit
method carries out the optimization process and trains the best model on all data . Only after this process is complete will you have a fully trained model that can be meaningfully saved . If you need to track progress during optimization, consider using Optuna's built-in study persistence features or logging mechanisms instead of trying to save incomplete model states .
To learn more, see the sources I considered: [[1] Auto](https://nixtlaverse.nixtla.io/mlforecast/auto.html)
Appreciate the feedback! Feel free to ask me again with additional guidance or context.
s
Hello @Max (Nixtla) I appreciate the bot's answer but it should simply say there isnt a way to do so perhaps? Or something like oh you cant save it like so because the trained model is not accessible and is temp elsewhere
m
Yes I agree. Lets see what the "AI" says.
@Ask Inkeep: Shouldn't you simply say there isn't a way to save the model as originally requested? Or something like oh you cant save it like so because the trained model is not accessible and is temp elsewhere
a
Thanks for tagging me @Max (Nixtla). I'll look through relevant sources and get back to you shortly.
You're correct. For the ​*MLForecast*​ library's 
AutoMLForecast
class, there isn't a way to save the model during the optimization process as you've attempted (1) . The trained models aren't accessible during the callback execution because the optimization process hasn't completed yet . The 
AutoMLForecast.save
method only works after the entire 
fit
process is finished . During optimization, the individual trial models are temporary and not stored in the main 
AutoMLForecast
object until the process completes and the best model is retrained on all data .
To learn more, see the sources I considered: [[1] Auto](https://nixtlaverse.nixtla.io/mlforecast/auto.html)
s
so no solution to store interim best config found as automl is running?
m
I think its time to bring the actual inteligent beings : @Olivier or @José Morales can you help here?
o
I didn't reply yet because I don't know the answer either 🙂
m
In that case, let's revert to first principles. @Sarim Zafar, what is the final outcome you are trying to achieve? Maybe we can find an alternative solution.
3 Views
Open in Slack
PreviousNext
Powered by