Hi everyone, I am facing an issue training all kinds of

Auto

models. It starts training and gets stuck, and never recovers. I am using Python 3.10.1 on Ubuntu 20.04.6 LTS.

In [1]: from neuralforecast import NeuralForecast                                                                                                     
                                                                                                                                                      
In [2]: from neuralforecast.auto import AutoNBEATS
                                                                           
In [3]: from neuralforecast.utils import AirPassengersDF
   ...:                                                                                                                                               
   ...: Y_df = AirPassengersDF
   ...: Y_df.head()
Out[3]: 
   unique_id         ds      y
0        1.0 1949-01-31  112.0
1        1.0 1949-02-28  118.0
2        1.0 1949-03-31  132.0
3        1.0 1949-04-30  129.0
4        1.0 1949-05-31  121.0

In [4]: models = [AutoNBEATS(h=12)]

In [5]: nf = NeuralForecast(models=models, freq='M')

In [6]: nf.fit(df=Y_df)
2023-09-25 18:10:00,972 INFO worker.py:1621 -- Started a local Ray instance.
2023-09-25 18:10:01,764 INFO tune.py:226 -- Initializing Ray automatically. For cluster usage or custom Ray initialization, call `ray.init(...)` befor
e `Tuner(...)`.
2023-09-25 18:10:01,767 INFO tune.py:666 -- [output] This will use the new output engine with verbosity 0. To disable the new output and use the legac
y output engine, set the environment variable RAY_AIR_NEW_OUTPUT=0. For more information, please see <https://github.com/ray-project/ray/issues/36949>
+--------------------------------------------------------------------+
| Configuration for experiment     _train_tune_2023-09-25_18-09-58   |
+--------------------------------------------------------------------+
| Search algorithm                 BasicVariantGenerator             |
| Scheduler                        FIFOScheduler                     |
| Number of trials                 10                                |
+--------------------------------------------------------------------+

View detailed results here: <$USER>/ray_results/_train_tune_2023-09-25_18-09-58
To visualize your results with TensorBoard, run: `tensorboard --logdir <$USER>/ray_results/_train_tune_2023-09-25_18-09-58`

(_train_tune pid=401980) Global seed set to 10
(_train_tune pid=401980) [rank: 0] Global seed set to 10
(_train_tune pid=401980) Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/3

It does not progress beyond this point. Issuing

$ ps aux | grep ray

points out to multiple

ray::IDLE

processes. Any help regarding this would be great! Thanks.

Hey @Sayan Sinha Can you try to limit the gpus to only one?

Thanks @Kin Gtz. Olivares, that seems to help! However, now I am coming across a Unicode encoding error:

UnicodeEncodeError: 'latin-1' codec can't encode characters in position 74-83: ordinal not in range(256)

Awesome @Sayan Sinha!!

Sure!

Thanks

here you go: https://github.com/Nixtla/neuralforecast/issues/760