Brian Head
09/13/2023, 8:51 PMKevin Kho
09/13/2023, 10:00 PMbackend
object like this and the backend has the cross validate methodBrian Head
09/13/2023, 11:07 PMcross_validation
but that just creates the a spark datframe itself. But doesn't run/train the models.
See example here:
**
sf = StatsForecast(
models=SF_models,
freq='M',
# n_jobs = -1,
fallback_model = Naive()
)
SF_crossvalidation_df = sf.cross_validation(df = sdf,
h = 3,
step_size = 1,
n_windows = 5)SF_crossvalidation_df = sf.cross_validation(df = sdf,
h = 3,
step_size = 1,
n_windows = 5)
**
Also tried the parallel=backend
in the link you provided in both the StatsForecast
and cross_validation
. When I put it in the former I get an error "TypeError: __init__() got an unexpected keyword argument 'parallel'". When I put it in the latter I get an error "TypeError: cross_validation() got an unexpected keyword argument 'parallel'".Kevin Kho
09/13/2023, 11:15 PMbackend = FugueBackend(spark, {"fugue.spark.use_pandas_udf":True})
backend.cross_validation(df = sdf,
h = 3,
step_size = 1,
n_windows = 5)
José Morales
09/14/2023, 1:51 AMcv_results = sf.cross_validation(df=spark_df, h=10)
cv_results.write.parquet('cv_results')
If you're using a remote cluster make sure to save it in a shared storage like s3Brian Head
09/14/2023, 5:58 PMKevin Kho
09/14/2023, 6:05 PM