Yes, spark is great. We use fugue as a backend that connects with spark, ray and dask. Is a great tool. @Han Wang and @Kevin Kho have been great allies in our mission. :)