Hi team. I am trying to use the Spark implementation of distributed computing. To import I use the following

from mlforecast.distributed import DistributedMLForecast

and

from mlforecast.distributed.models.spark.lgb import SparkLGBMForecast

. I have installed the library on the cluster I am using (I am using Databricks) but I get an

ImportError

and I cannot figure out if I am using the library wrong or something is wrong with my cluster.

ImportError: cannot import name 'DistributedMLForecast' from 'mlforecast.distributed' (/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/mlforecast/distributed/__init__.py)

Hey. Which version do you have installed? Can you run:

import mlforecast
print(mlforecast.__version__)

Hey! I accidently pinned the version to

0.2.0

but I have changed that to

0.12.0

as per the mlforecast page on pypi. However, now I am getting

ModuleNotFoundError

issues when importing. First it says

No module named 'fugue'

when I then try to install that I get

No module named 'synapse'

when I then install that I get

No module named '<http://synapse.ml|synapse.ml>'

and I haven't tried to install further - is this the intended way? Or should I have installed the libraries on the cluster explicitly like with

mlforecast

?

Yes, you need them as cluster libraries as well, which you can also get by adding the spark extra, i.e.

mlforecast[spark]

in your install command. SynapseML is a bit weird though, since you need to install the scala package which somehow enables the python package. I recommend following the installation instructions here