Hi Team, I am trying to do an evaluation of my for...
# general
t
Hi Team, I am trying to do an evaluation of my forecasts with the
utilsforecast.evaluation.evaluate
function but I keep hitting the following error when adding the train_df parameter. I’ve had a look at the source code which indicates that train_df is optional, however if I omit the train_df I get the same error. call:
Copy code
from utilsforecast.evaluation import evaluate
from utilsforecast.losses import (
    mse,   # mean square error
    mape,  # mean absolute percentage error
    mae,   # mean absolute error
    mase,  # mean absolute scaled error
    rmse,  # root mean square error
    mqloss, # multi-quantile loss
    scaled_crps, # scaled continues ranked probability score
    )
eval_df = evaluate(forecasts, metrics=[metrics], train_df=train)
> File “/Users/.pyenv/versions/datascience/lib/python3.11/site-packages/utilsforecast/evaluation.py”, line 72, in evaluate > metric_requires_y_train = { > ^ > File “/Users/.pyenv/versions/datascience/lib/python3.11/site-packages/utilsforecast/evaluation.py”, line 73, in dictcomp > _function_name(m): “train_df” in inspect.signature(m).parameters > ^^^^^^^^^^^^^^^^^ > File “/Users/.pyenv/versions/datascience/lib/python3.11/site-packages/utilsforecast/evaluation.py”, line 22, in _function_name > name = f.name > ^^^^^^^^^^ > AttributeError: ‘list’ object has no attribute ‘__name__‘. Did you mean: ‘__ne__‘? my forecasts dataframe has the following columns:
unique_id
ds
Naive
Naive-lo-95
Naive-hi-95
MSTL
MSTL-lo-95
MSTL-hi-95
AutoETS
AutoETS-lo-95
AutoETS-hi-95
combined-lo
combined-hi
combined-base
my train dataframe has the following columns:
ds
y
unique_id
and the metrics list is simply [mse, rmse, mape] Any assistance would be appreciated. Thanks in advance.
m
Hello! I think your forecasts dataframe must include the actual values. By default, it looks for a column called
y
. Otherwise, you can specify the column name using
target_col
. I created a minimal example to mimic your situation and it works on my end:
Copy code
data = {
    'unique_id': [1, 1],  # Setting constant value to 1
    'ds': ['2024-01-01', '2024-01-02'],
    'y': [102, 103],
    'Naive': [100, 105],
    'Naive-lo-95': [95, 100],
    'Naive-hi-95': [105, 110],
    'MSTL': [102, 107],
    'MSTL-lo-95': [97, 102],
    'MSTL-hi-95': [107, 112],
    'AutoETS': [103, 108],
    'AutoETS-lo-95': [98, 103],
    'AutoETS-hi-95': [108, 113],
    'combined-lo': [96, 101],
    'combined-hi': [106, 111],
    'combined-base': [101, 106]
}

# Creating the DataFrame
df_sample = pd.DataFrame(data)

evaluation_df = evaluate(df=df_sample, metrics=[mse, rmse, mape])
I hope this helps!
t
Thank you for your rep[y I will try that.
I ran your exact example and was still hitting the same error. I realised that I was on an older version of utilsforecast (0.0.24) I have upgraded to the latest version (0.1.2) and re-run and it is working. Thanks
1