Is there a way to optimize models during training on CRPS? Here I saw how you could tweak the model for MAE: https://github.com/Nixtla/mlforecast/discussions/430. What function could we use to make a probabilistic loss work?

for examle in lightgbm you just change the objective function in our parameter spacee to quantile and then add one more multiple quantiles:

import lightgbm as lgb

params = {
    'objective': 'quantile',
    'alpha': 0.5,  # 0.5 corresponds to the median (pinball loss at 50% quantile)
    'learning_rate': 0.1,
    'num_leaves': 31,
    'metric': 'quantile'
}

# Create dataset
train_data = lgb.Dataset(X_train, label=y_train)

# Train model
model = lgb.train(params, train_data, num_boost_round=100)

I am using automodels. So optimizing automodels for MAE goes as follows, I want to optimize on CRPS as a metric:

def loss_fn(df, train_df): return mae(df, models=['model'])['model'].mean()

auto_mlf_large.fit(
    grote_agg_df, 
    n_windows=n_windows, 
    h=h, 
    step_size=step_size, 
    fitted=True, 
    num_samples=40, 
    loss=loss_fn
)

but when you define your search space cant you add objective? so for example this is from the documentation. objective means the loss function and here l1, l2 and mape are tested during tuning. you can also set just one value here and then it will always be selected (and basically the normal objective function will be overwritten).

def my_lgb_config(trial: optuna.Trial):
    return {
        'learning_rate': 0.05,
        'verbosity': -1,
        'num_leaves': trial.suggest_int('num_leaves', 2, 128, log=True),
        'objective': trial.suggest_categorical('objective', ['l1', 'l2', 'mape']),
    }

my_lgb = AutoModel(
    model=lgb.LGBMRegressor(),
    config=my_lgb_config,
)
auto_mlf = AutoMLForecast(
    models={'my_lgb': my_lgb},
    freq=1,
    season_length=24,
).fit(
    train,
    n_windows=2,
    h=horizon,
    num_samples=2,
)
preds = auto_mlf.predict(horizon)
evaluate(preds, group)

I am using the default optimization also provided in the same documentation. This tunes both model parameters and features if I am correct. The example. code:

optuna.logging.set_verbosity(optuna.logging.ERROR)
auto_mlf = AutoMLForecast(
    models={'lgb': AutoLightGBM(), 'ridge': AutoRidge()},
    freq=1,
    season_length=24,
)
auto_mlf.fit(
    train,
    n_windows=2,
    h=horizon,
    num_samples=2,  # number of trials to run
)

I don’t think so. There are no features specified. Which features should that be? You could build thousands of oh features from lags and rolling windows. This is just the most basic example.

This is what the top configuration looks like: {'lgb': {'model_params': {'bagging_freq': 1, 'learning_rate': 0.05, 'verbosity': -1, 'n_estimators': 585, 'lambda_l1': 0.0007029873577904313, 'lambda_l2': 3.633333734134197e-05, 'num_leaves': 133, 'feature_fraction': 0.9149810345204117, 'bagging_fraction': 0.6354647629368543, 'objective': 'l1'}, 'mlf_init_params': {'lags': None, 'target_transforms': [<mlforecast.target_transforms.GlobalSklearnTransformer at 0x7fde7264aad0>, <mlforecast.target_transforms.Differences at 0x7fde7174bb10>, <mlforecast.target_transforms.LocalStandardScaler at 0x7fde7174b610>], 'lag_transforms': {1: [ExponentiallyWeightedMean(alpha=0.9)], 12: [RollingMean(min_samples=1, window_size=12)]}, 'date_features': None, 'num_threads': 1}, 'mlf_fit_params': {'static_features': ['unique_id']}}

i am not sure where is coming from. but you can see that there are no lags used and no date features used. so i would say this is not optimal. but you are right, it seems there are some default search spaces that are used based on the length of the seasonal period. But it might make sense to reveiw this and add your own ideas also. I also saw that you are using season_legnth=24, which means you would be having hourly data, but i guess you are more likely having weekly (season_length=52) and monthly (season_length=12) data. Maybe try changing your season length and check results again. also about the use of date features I am not sure if this is covered by automl, because your output shows no date features are used.

My data is monthly, so I set

season_length=12

. I already added date features myself (month, Fourier terms) in the dataset. I’ll add my own lags and check again.

you added date features? but it looks like no date features are used:

'date_features': None,

I added features to my datafile itself as exogenous variables. I just looked at the model configuration for xgb and that one looks like the following, which does have lags and date features however performs a little worse then lightgbm which has no lags and no date features according to the configuration: 'model_params': {'n_estimators': 163, 'max_depth': 10, 'learning_rate': 0.017421795248813172, 'subsample': 0.8768126003056615, 'colsample_bytree': 0.6365489579310208, 'reg_lambda': 0.17045414030974484, 'reg_alpha': 1.3436810645082065e-05, 'min_child_weight': 3}, 'mlf_init_params': {'lags': [12], 'target_transforms': [<mlforecast.target_transforms.GlobalSklearnTransformer at 0x7f69adb59d10>, <mlforecast.target_transforms.Differences at 0x7f69adaff950>, <mlforecast.target_transforms.LocalStandardScaler at 0x7f6b4939f850>], 'lag_transforms': {1: [ExponentiallyWeightedMean(alpha=0.9)], 12: [RollingMean(min_samples=1, window_size=12)]}, 'date_features': ['month', 'year'], 'num_threads': 1}, 'mlf_fit_params': {'static_features': ['unique_id']}}

can you show your full settings. what is your h, step_size, n_windows?

h = 18 step_size = 1 n_windows = 10

looks fine