Hi guys, a quick question/suggestion. In

mlforecast

in

fcst.preprocess()

, the default value for

dropna

is True, so in cases when we have intermittent series and we generate features like

rolling_std

, NAs might occur anywhere in data (not just at initial rows) and we are dropping all those observations. the consequence might be inaccurate CV estimate when we try to evaluate the model. May be a warning in the function to notify the user about the default value of

dropna

?

Thanks for the comment. Cc @José Morales

Hey @Arsa Nikzad. I think that if you have any null values in your target you get an error when trying to run the preprocess because all transformations propagate the null values, were you able to run it with missing values in the middle?

Hi @José Morales. thanks for the reply. the target does not have null values. the issue is with

rolling_std

when we have a set of consecutive zeros larger than window size in target. the root cause of issue seems to be

_rolling_std

in

window_ops.rolling

where it generates large negative numbers in above situation and these negative numbers are then converted to NAN. attached is an example.

data = pd.DataFrame({
    'date': pd.date_range(start='2019-01-01', end='2020-12-31', freq='MS'),
    'sprid': 1.,
    'target': [1., 2., 0., 4., 0., 0., 0., 0., 9., 10., 11., 12] * 2
})

models = [lgb.LGBMRegressor(**{})]
fcst = MLForecast(
    models=models,
    freq='MS',
    lags=[1],
    lag_transforms={
        1: [(rolling_std, 3)]
    }
)

preprocessed_df = fcst.preprocess(data, id_col='sprid', time_col='date', target_col='target', dropna=False)
print(preprocessed_df)

## check _rolling_std
from window_ops.rolling import  _rolling_std
a = np.array([1, 2, 0, 4, 0, 0, 0, 0, 9, 10, 11, 12] * 2)
print(_rolling_std(a, 3))

Thanks for the example! I think a warning for these cases is definitely useful. Will add it soon

additionally,

rolling_std

should generate zeros instead of NAN for these cases.

Hi @Arsa Nikzad! Happy to work on that. Would you mind opening an issue for that on GH. Here is the link.

Sure, will do!

Hey @Arsa Nikzad, we just pushed window-ops 0.0.14 which should fix this issue with the rolling_std

Thank you.