Hi Nixtla Team Thanks for developing a wonderful library for

Question

Hi Nixtla Team Thanks for developing a wonderful library for forecasting I am getting an error when I try to fit AutoLSTM and AutoNHITS The error is as follows ``` gt 168 raise RuntimeError error msg 169 170 return self trial to result best trial ``` RuntimeError No best trial found for the given metric loss This means that no trial has reported this metric or all values reported for this metric are NaN To not ignore NaN values you can set the `filter nan and inf` arg to False My config looks like this capture from ray import tune config nhits = input size tune choice 4 8 Length of input window start padding enabled True n blocks 3 1 Length of input window activation relu mlp units 3 3 3 Length of input window n pool kernel size tune choice 3 1 3 2 3 4 8 4 2 1 1 MaxPooling Kernel size n freq downsample tune choice 8 4 2 1 1 1 1 1 1 1 Interpolation expressivity ratios learning rate tune loguniform 1e 4 1e 2 Initial Learning rate scaler type tune choice robust standard Scaler type max steps tune choice 500 1000 Max number of training iterations batch size tune choice 1 2 4 8 16 32 64 128 Number of series in batch windows batch size tune choice 1 2 4 8 16 32 64 128 Number of windows in batch stack types trend seasonality random seed tune randint 1 20 Random seed hist exog list VOICE TRAFFIC SUCCESSFULL CALLS config lstm = input size tune choice 4 8 Length of input window encoder hidden size tune choice 4 8 16 32 64 128 Hidden size of LSTM cells encoder activation tune choice relu encoder n layers tune choice 1 2 3 Number of layers in LSTM learning rate tune loguniform 1e 4 1e 2 Initial Learning rate scaler type tune choice robust standard Scaler type max steps tune choice 100 200 Max number of training iterations batch size tune choice 2 4 8 16 32 64 128 Number of series in batch random seed tune randint 1 20 Random seed hist exog list VOICE TRAFFIC SUCCESSFULL CALLS config nbeatsx = input size tune choice 4 8 Length of input window learning rate tune loguniform 1e 4 1e 2 scaler type tune choice robust standard max steps tune choice 100 200 n polynomials 0 n blocks 1 1 stack types trend seasonality dropout prob theta 0 5 mlp units 4 4 8 8 16 16 32 32 random seed tune randint 1 20 hist exog list VOICE TRAFFIC SUCCESSFULL CALLS config tft = input size tune choice 4 8 hidden size tune choice 8 16 learning rate tune loguniform 1e 4 1e 2 scaler type tune choice robust max steps tune choice 250 500 batch size tune choice 8 16 random seed tune randint 1 20 hist exog list VOICE TRAFFIC SUCCESSFULL CALLS My Prediction Code looks like this import numpy as np import pandas as pd import pytorch lightning as pl import matplotlib pyplot as plt from neuralforecast import NeuralForecast from neuralforecast auto import AutoLSTM AutoNHITS AutoTFT AutoNBEATSx from neuralforecast losses pytorch import MQLoss MAE HuberMQLoss from ray tune search hyperopt import HyperOptSearch levels= 80 90 h = Y test df ds nunique search alg=HyperOptSearch metric= loss mode= min modelNHITS = AutoNHITS h=h loss=MAE config=config nhits num samples=50 cpus=1 modelLSTM = AutoLSTM h=h loss=MAE config=config lstm num samples=50 cpus=1 modelNBEATSx = AutoNBEATSx h=h config=config nbeatsx loss=MAE valid loss=MAE num samples=50 cpus=1 modelTFT = AutoTFT h=h config=config tft loss=MAE valid loss=MAE num samples=50 cpus=1 nf = NeuralForecast models= modelNHITS modelLSTM freq= W nf fit df=Y train df Y hat df = nf predict Y hat df = nf predict insample step size=h Y hat df = Y hat df reset index drop=False drop columns= unique id ds plot df = pd concat Y test df Y hat df axis=1 plot df = pd concat Y train df plot df plot df = plot df plot df unique id== WEEKLY drop unique id axis=1 plt plot plot df ds plot df y c= black label= Actual plt plot plot df ds plot df AutoNHITS c= blue label= NHITS Prediction plt plot plot df ds plot df AutoLSTM c= purple label= LSTM Prediction plt plot plot df ds plot df AutoTFT median c= green label= AutoTFT Prediction plt plot plot df ds plot df AutoNBEATSx c= Green label= NBeatsx Prediction plt fill between x=plot df ds 40 y1=plot df AutoLSTM lo 90 40 values y2=plot df AutoLSTM hi 90 40 values alpha=0 4 label= level 90 plt legend plt grid plt plot plt figure figsize= 10 5 plt plot Y hat df ds Y hat df y label= True plt plot Y hat df ds Y hat df AutoNHITS c= blue label= NHITS Forecast plt plot Y hat df ds Y hat df AutoLSTM c= purple label= LSTM Forecast plt plot Y hat df ds Y hat df AutoNBEATSx c= green label= NBeatsx Prediction plt axvline Y hat df ds iloc 20 color= black linestyle= label= Train Test Split plt xlabel Timestamp t plt ylabel Demand for Voice Services Call Attempts plt grid plt legend Can someone please guide me what I am doing wrong here Any help guidance will help me

Cristian (Nixtla) · Answer

Hi < Naveen Chandra> Tune wont stop if there is a bug for each individual run it will simply go to the next The error you see occurs when all individual `num samples` runs failed I suggest starting simpler with much smaller configs and less `num samples` 3 or 5 There are some issues with the config for instance the `nhits` only have `identity` blocks no trend or seasonality Also the numbers of elements in `n pool kernel size` needs to match the number of stacks length of `stack types`

Cristian (Nixtla) · Answer

here is a shorter config that should work ```config nhits = input size tune choice 4 8 Length of input window start padding enabled True stack types 3 identity n blocks 3 1 Length of input window mlp units 3 128 128 Length of input window n pool kernel size tune choice 3 1 3 2 3 4 MaxPooling Kernel size n freq downsample tune choice 4 2 1 1 1 1 Interpolation expressivity ratios learning rate tune loguniform 1e 4 1e 2 Initial Learning rate scaler type tune choice robust standard Scaler type max steps tune choice 500 1000 Max number of training iterations batch size tune choice 32 64 128 Number of series in batch windows batch size tune choice 64 128 256 Number of windows in batch random seed tune randint 1 20 Random seed ```

Cristian (Nixtla) · Answer

Some other tips `mlp units` of 3 is extremely small I changed it to 128 `batch size` and `windows batch size` of 1 are also too small Let me know if the config above works Try with `num samples=2` while debugging

Naveen Chandra · Answer

Hi Crisitan Thanks for such a quick response I am getting this error which I have no idea how to fix with the config I mentioned Any pointers around why am I getting this error RuntimeError No best trial found for the given metric loss This means that no trial has reported this metric or all values reported for this metric are NaN To not ignore NaN values you can set the `filter nan and inf` arg to False

Cristian (Nixtla) · Answer

The error you see occurs when all individual `num samples` runs failed or have a NaN loss

Cristian (Nixtla) · Answer

can you try with the config I sent you

Naveen Chandra · Answer

< Cristian Nixtla > Thanks for the config which solved my problem Really appreciate your quick responses Any ideas suggestion as to which specific config might be causing this error so that I can avoid making same mistake in future

Naveen Chandra · Answer

Also I am getting the same issue for LSTM Config as well A quick note I am training a very small dataset of 101 time samples of weekly data Hence I was keeping the mlp units and other config small My LSTM Config config lstm = input size tune choice 4 8 Length of input window encoder hidden size tune choice 4 8 16 32 64 128 Hidden size of LSTM cells encoder activation tune choice relu encoder n layers tune choice 1 2 3 Number of layers in LSTM learning rate tune loguniform 1e 4 1e 2 Initial Learning rate scaler type tune choice robust standard Scaler type max steps tune choice 100 200 Max number of training iterations batch size tune choice 2 4 8 16 32 64 128 Number of series in batch random seed tune randint 1 20 Random seed hist exog list VOICE TRAFFIC SUCCESSFULL CALLS