Can anyone tell me how to or direct me to documentation on how to do cross-validation while using spark for distributed processing? The page linked below says it's possible, but I can't find any documentation or tutorials on how to. Also couldn't find any previous posts with this question. https://nixtla.github.io/statsforecast/docs/getting-started/getting_started_complete.html#train-multiple-models-for-many-series

This you try just passing a Spark DataFrame instead? I forget if that works already

@Kevin Kho thanks for your response. I tried passing the spark dataframe to

cross_validation

but that just creates the a spark datframe itself. But doesn't run/train the models. See example here: ** sf = StatsForecast( models=SF_models, freq='M', # n_jobs = -1, fallback_model = Naive() ) SF_crossvalidation_df = sf.cross_validation(df = sdf, h = 3, step_size = 1, n_windows = 5)SF_crossvalidation_df = sf.cross_validation(df = sdf, h = 3, step_size = 1, n_windows = 5) ** Also tried the

parallel=backend

in the link you provided in both the

StatsForecast

and

cross_validation

. When I put it in the former I get an error "TypeError: __init__() got an unexpected keyword argument 'parallel'". When I put it in the latter I get an error "TypeError: cross_validation() got an unexpected keyword argument 'parallel'".

Ah ok. I meant try:

backend = FugueBackend(spark, {"fugue.spark.use_pandas_udf":True})
backend.cross_validation(df = sdf,
                          h = 3,
                          step_size = 1,
                          n_windows = 5)

The cross_validation method returns a spark dataframe. As Kevin said, in order to trigger the action you have to do something with it, for example:

cv_results = sf.cross_validation(df=spark_df, h=10)
cv_results.write.parquet('cv_results')

If you're using a remote cluster make sure to save it in a shared storage like s3

Thank you @Kevin Kho and @José Morales this resolved my issues with statsforecast cv. Appreciate the help! I am having an issue with doing CV in MLforecast now. I can post my question on that in the mlforecast channel.

Nice!