This message was deleted.

So you are running on Windows? I don't use Windows but I know Ray has better support for Linux and Mac and has had issues with Windows in the past although it might not be related to your case and I don't know if you have access to Linux to test it there. Where is Ray trying to sync the checkpoints to? Some local directory on your PC? And is this a blocker for you? Does the training finish and you are able to predict regardless of that error or is that raising an exception?

Thanks for the reply. I was using windows and after several chatgpt sessions I realized the error might be coming from the windows file system format. I installed WSL on windows and it is working now when running WSL ubuntu.

Glad it worked out. Ray is a fantastic framework for parallel processing specially for DL/ML applications and it's a great feature of Nixtla libraries to take advantage of it but Ray like most other Python libararies runs the best on Linux. Even Mac is not great anymore after Apple introduced its own chipsets (M1, M2,...).

@windwine hi, could you share the guide you follow to install/configure GPU with the ubuntu WSL? I installed by following this https://docs.nvidia.com/cuda/wsl-user-guide/index.html But my conda or python environment in the ubuntu WSL still couldn't detect the GPU.

I followed https://docs.nvidia.com/cuda/wsl-user-guide/index.html and use the option1

I notice my wsl ubuntu is 18.04 while the CUDA Toolkit only provides 20.04, maybe that could be the reason why

yeah, just get a fresh wsl to be safe.

Also if you are using VSCode then you should be able to open Ray dashboard (Neuralforecast installs Ray as a dependency) and see all GPU and CPU cores during the training. VSCode automatically forwards ports so you can open the dashboard port in your browser. That's another way of seeing how many cores are detected and what they are doing.