Luis Huergo
08/24/2023, 7:00 PMLuis Huergo
08/24/2023, 8:49 PMPhil
08/24/2023, 9:27 PMnf2 = NeuralForecast.load(path='../cluster_results/results/b79f6fd0c50099a7519da800805bf436d4f4f4a6/')
RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.
I'm trying to debug this error on my own. I found this stackoverflow post. https://stackoverflow.com/questions/56369030/runtimeerror-attempting-to-deserialize-object-on-a-cuda-device
But I think this needs to be handle as part of the load function?https://github.com/Nixtla/neuralforecast/blob/1d78f503ac4cd0fdfa7aae9156a876fe94eb1db4/neuralforecast/core.py#L665
I think this can be solved if you have an extra argument to the load function map_location
MODEL_FILENAME_DICT[model_name].load_from_checkpoint(f"{path}/{model}", map_location=map_location)
If I specify map_location=torch.device('cpu'))
, it worksStefan Wiegand
08/25/2023, 12:12 PMAda Puślecka
08/25/2023, 12:19 PMtorchmetrics.metric.Metric
and thinking if it would be possible to use that.허성민
08/28/2023, 1:37 AMStefan Wiegand
08/28/2023, 3:28 PMlobbie lobbie
08/28/2023, 11:13 PMStefan Wiegand
08/30/2023, 1:03 PMMarco Zucchini
08/30/2023, 8:34 PMnf.cross_validation(df=Y_train_df, n_windows=10,step_size=1,val_size=horizon,sort_df=True,verbose=True)
`ValueError: You are trying to merge on datetime64[s] and object columns. If you wish to proceed you should use pd.concat`Actually I was not trying to merge anything at that point: is the method expecting another data format for 'ds' different from the simple fit? Thank you!Stefan Wiegand
08/31/2023, 7:34 PMFarzad E
08/31/2023, 8:50 PMOut of bounds nanosecond timestamp: 2385-12-31 00:00:00
when I do nf.predict(futr_df=df_test). But my df_test only has 364 days in it with dates going from 2022-01-03 to 2023-01-01. There is no date even close to 2385! Also, the 'ds' column is properly converted to datetime type. It's weird. I am thinking that 2385 might be coming from adding a date to the last point in an input batch? I'm not sure. There is a line in the error stack pointing to core.py line 109 dates=np.tile(last_date_f(last_dates[0], len(dataset)).
Has anyone seen this type of error before?Raul Gherman
09/03/2023, 11:34 AMAfiq Johari
09/05/2023, 8:28 AMmarah othman
09/05/2023, 8:31 AMBWBarber
09/08/2023, 2:33 AMY_df['noise'] = np.random.rand(len(Y_df))
# split to train and test
Y_train_df = Y_df[:-horizon]
Y_test_df = Y_df[-horizon:]
# Instantiate HINT
# BaseNetwork + Distribution + Reconciliation
nhits = NHITS(h=horizon,
input_size=24,
loss=GMM(n_components=10, level=level),
hist_exog_list=['month'],
futr_exog_list=['noise'],
max_steps=2000,
early_stop_patience_steps=10,
val_check_steps=50,
scaler_type='robust',
learning_rate=1e-3,
valid_loss=sCRPS(level=level))
model = HINT(h=horizon, S=S_df.values,
model=nhits, reconciliation='BottomUp')
# Fit and Predict
nf = NeuralForecast(models=[model], freq='MS')
nf.fit(df=Y_train_df, val_size=12)
Y_hat_df = nf.predict(Y_test_df)
When I run this code, I receive the error: "RuntimeError: normal expects all elements of std >= 0.0"
Has this been seen before? Not sure exactly what's going on. Everything works fine when I comment out futr_exog_listFarzad E
09/08/2023, 4:29 PMwindwine
09/11/2023, 4:42 PMAfiq Johari
09/13/2023, 8:13 AMstorage_path
for the AutoNHITS
or HyperOptSearch
?
I'm trying to get the ray
or tune
working with neuralforecast
and I keep getting this error below.
(_train_tune pid=21000) 2023-09-13 08:02:48,610 ERROR syncer.py:466 -- Caught sync error: Sync process failed: GetFileInfo() yielded path 'C:/Users/bubu/ray_results/_train_tune_2023-09-13_08-02-34/_train_tune_7a885f1f_1_h=12,input_size=24,learning_rate=0.0032,loss=ref_ph_de895953,max_steps=10,n_freq_downsample=24_12_1,n_pool__2023-09-13_08-02-39/error.pkl', which is outside base dir 'C:\Users\bubu\ray_results\_train_tune_2023-09-13_08-02-34\_train_tune_7a885f1f_1_h=12,input_size=24,learning_rate=0.0032,loss=ref_ph_de895953,max_steps=10,n_freq_downsample=24_12_1,n_pool__2023-09-13_08-02-39\'. Retrying after sleeping for 1.0 seconds...
(_train_tune pid=21000) 2023-09-13 08:02:49,617 ERROR syncer.py:466 -- Caught sync error: Sync process failed: GetFileInfo() yielded path 'C:/Users/bubu/ray_results/_train_tune_2023-09-13_08-02-34/_train_tune_7a885f1f_1_h=12,input_size=24,learning_rate=0.0032,loss=ref_ph_de895953,max_steps=10,n_freq_downsample=24_12_1,n_pool__2023-09-13_08-02-39/error.pkl', which is outside base dir 'C:\Users\bubu\ray_results\_train_tune_2023-09-13_08-02-34\_train_tune_7a885f1f_1_h=12,input_size=24,learning_rate=0.0032,loss=ref_ph_de895953,max_steps=10,n_freq_downsample=24_12_1,n_pool__2023-09-13_08-02-39\'. Retrying after sleeping for 1.0 seconds...
It seems this is a known bug and a suggested workaround is to predefine the storage path as below https://github.com/ray-project/ray/issues/36448#issuecomment-1673274452
from ray import air, tune
tuner = tune.Tuner(
train_fn,
run_config=air.RunConfig(storage_path="./")
)
tuner.fit()
However, I'm not sure where to define run_config=air.RunConfig(storage_path="./")
with AutoNHITS
Really appreciate any tips on this issue, thanks!Naveen Chandra
09/14/2023, 4:13 PM--> 168 raise RuntimeError(error_msg)
169
170 return self._trial_to_result(best_trial)
RuntimeError: No best trial found for the given metric: loss. This means that no trial has reported this metric, or all values reported for this metric are NaN. To not ignore NaN values, you can set the filter_nan_and_inf
arg to False."
My config looks like this:
%%capture
from ray import tune
config_nhits = {
"input_size": tune.choice([4, 8]), # Length of input window
"start_padding_enabled": True,
"n_blocks": 3*[1], # Length of input window
"activation": "relu",
"mlp_units": 3 * [[3, 3]], # Length of input window
"n_pool_kernel_size": tune.choice([3*[1], 3*[2], 3*[4],
[8, 4, 2, 1, 1]]), # MaxPooling Kernel size
"n_freq_downsample": tune.choice([[8, 4, 2, 1, 1],
[1, 1, 1, 1, 1]]), # Interpolation expressivity ratios
"learning_rate": tune.loguniform(1e-4, 1e-2), # Initial Learning rate
"scaler_type": tune.choice(['robust', 'standard']), # Scaler type
"max_steps": tune.choice([500, 1000]), # Max number of training iterations
"batch_size": tune.choice([1,2,4,8,16,32,64,128]), # Number of series in batch
"windows_batch_size": tune.choice([1,2,4,8,16,32,64,128]), # Number of windows in batch
"stack_types": ['trend', 'seasonality'],
"random_seed": tune.randint(1, 20), # Random seed
"hist_exog_list": ['VOICE_TRAFFIC','SUCCESSFULL_CALLS']
}
config_lstm = {
"input_size": tune.choice([4,8]), # Length of input window
"encoder_hidden_size": tune.choice([4,8,16,32,64,128]), # Hidden size of LSTM cells
"encoder_activation": tune.choice(['relu']),
"encoder_n_layers": tune.choice([1,2,3]), # Number of layers in LSTM
"learning_rate": tune.loguniform(1e-4, 1e-2), # Initial Learning rate
"scaler_type": tune.choice(['robust', 'standard']), # Scaler type
"max_steps": tune.choice([100, 200]), # Max number of training iterations
"batch_size": tune.choice([2,4,8,16,32,64,128]), # Number of series in batch
"random_seed": tune.randint(1, 20), # Random seed
"hist_exog_list": ['VOICE_TRAFFIC','SUCCESSFULL_CALLS']
}
config_nbeatsx = {
"input_size": tune.choice([4,8]), # Length of input window
"learning_rate": tune.loguniform(1e-4, 1e-2),
"scaler_type": tune.choice(['robust', 'standard']),
"max_steps": tune.choice([100, 200]),
"n_polynomials": 0,
"n_blocks": [1,1],
"stack_types": ['trend', 'seasonality'],
"dropout_prob_theta": 0.5,
"mlp_units": [[4,4],[8,8],[16,16],[32,32]],
"random_seed": tune.randint(1, 20),
"hist_exog_list": ['VOICE_TRAFFIC','SUCCESSFULL_CALLS']
}
#config_tft = {
# "input_size": tune.choice([4,8]),
# "hidden_size": tune.choice([8, 16]),
# "learning_rate": tune.loguniform(1e-4, 1e-2),
# "scaler_type": tune.choice(['robust']),
# "max_steps": tune.choice([250, 500]),
# "batch_size": tune.choice([8, 16]),
# "random_seed": tune.randint(1, 20),
# "hist_exog_list": ['VOICE_TRAFFIC','SUCCESSFULL_CALLS']
My Prediction Code looks like this:
import numpy as np
import pandas as pd
import pytorch_lightning as pl
import matplotlib.pyplot as plt
from neuralforecast import NeuralForecast
from neuralforecast.auto import AutoLSTM,AutoNHITS,AutoTFT, AutoNBEATSx
from neuralforecast.losses.pytorch import MQLoss,MAE,HuberMQLoss
from ray.tune.search.hyperopt import HyperOptSearch
levels=[80,90]
h = Y_test_df['ds'].nunique()
#search_alg=HyperOptSearch(metric="loss", mode="min"),
modelNHITS = AutoNHITS(h=h, loss=MAE(), config=config_nhits, num_samples=50, cpus=1)
modelLSTM = AutoLSTM(h=h, loss=MAE(), config=config_lstm, num_samples=50, cpus=1)
#modelNBEATSx = AutoNBEATSx(h=h, config=config_nbeatsx, loss=MAE(),valid_loss=MAE(), num_samples=50, cpus=1)
#modelTFT = AutoTFT(h=h, config=config_tft, loss=MAE(),valid_loss=MAE(), num_samples=50, cpus=1)
nf = NeuralForecast(models=[modelNHITS,modelLSTM],freq='W')
nf.fit(df=Y_train_df)
#Y_hat_df = nf.predict()
Y_hat_df = nf.predict_insample(step_size=h)
#Y_hat_df = Y_hat_df.reset_index(drop=False).drop(columns=['unique_id','ds'])
#plot_df = pd.concat([Y_test_df, Y_hat_df], axis=1)
#plot_df = pd.concat([Y_train_df, plot_df])
#plot_df = plot_df[plot_df.unique_id=='WEEKLY'].drop('unique_id', axis=1)
#plt.plot(plot_df['ds'], plot_df['y'], c='black', label='Actual')
#plt.plot(plot_df['ds'], plot_df['AutoNHITS'], c='blue', label='NHITS-Prediction')
#plt.plot(plot_df['ds'], plot_df['AutoLSTM'], c='purple', label='LSTM-Prediction')
##plt.plot(plot_df['ds'], plot_df['AutoTFT-median'], c='green', label='AutoTFT-Prediction')
##plt.plot(plot_df['ds'], plot_df['AutoNBEATSx'], c='Green', label='NBeatsx-Prediction')
##plt.fill_between(x=plot_df['ds'][-40:],
## y1=plot_df['AutoLSTM-lo-90'][-40:].values,
## y2=plot_df['AutoLSTM-hi-90'][-40:].values,
## alpha=0.4, label='level 90')
#plt.legend()
#plt.grid()
#plt.plot()
plt.figure(figsize=(10, 5))
plt.plot(Y_hat_df['ds'], Y_hat_df['y'], label='True')
plt.plot(Y_hat_df['ds'], Y_hat_df['AutoNHITS'], c='blue', label='NHITS-Forecast')
plt.plot(Y_hat_df['ds'], Y_hat_df['AutoLSTM'], c='purple', label='LSTM-Forecast')
#plt.plot(Y_hat_df['ds'], Y_hat_df['AutoNBEATSx'], c='green', label='NBeatsx-Prediction')
plt.axvline(Y_hat_df['ds'].iloc[-20], color='black', linestyle='--', label='Train-Test Split')
plt.xlabel('Timestamp [t]')
plt.ylabel('Demand for Voice Services(Call Attempts)')
plt.grid()
plt.legend()
Can someone please guide me what I am doing wrong here? Any help/guidance will help me..J.
09/16/2023, 12:56 PMAlex Wang
09/19/2023, 9:45 PMTemporalNorm
uses one set of normalization per sequence (basically InstanceNorm for sequence data without learnable parameters)? I would have expected the offset and scale to be calculated over the entire training dataset and kept constant for every sequence @Cristian (Nixtla)GoostValley
09/20/2023, 11:59 AMaccelerator="cuda:0"
but errors were raised. Any solutions except setting "CUDA_VISIBLE_DIVICE" in the command line?Guillaume GALIE
09/21/2023, 1:21 PMGuillaume GALIE
09/21/2023, 2:31 PMJ.
09/22/2023, 12:57 PMinput_size=-1 && inference_input_size=-1
but they have loss is >0.0 so it must be something else.
Any help is appreciated!Phil
09/22/2023, 3:14 PMn_freq_downsample
and n_pool_kernel_size
. My question may also stem from how these parameters interact with one another.
From my understanding, the higher the value in n_freq_downsample
, the less datapoints we will have after downsampling. i.e. if we have daily data and we downsample with n_freq_downsample = 7
, we will have 1/7 the amount of data.
n_pool_kernel_size
also has a "downsampling effect", if we have an input size of N
and n_pool_kernel_size = 2
, then we will now have N / 2
datapoints.
First of all, is this understanding correct? And if so, it seems like we would want these two hyperparameters be inversely proportional. I would not want to downsample by a large number and then apply large kernel size pooling layer. These would dramatically decrease the amount of information being fed into my model?
The reason I am asking this question is because I was looking at the AutoNHITS parameter space
"n_pool_kernel_size": tune.choice(
[[2, 2, 1], 3 * [1], 3 * [2], 3 * [4], [8, 4, 1], [16, 8, 1]]
),
"n_freq_downsample": tune.choice(
[
[168, 24, 1],
[24, 12, 1],
[180, 60, 1],
[60, 8, 1],
[40, 20, 1],
[1, 1, 1],
]
),
Intuitively, I would have assumed that the last two choices for n_pool_kernel_size
would have been reversed. I.E. [1, 4, 8], [1, 8, 16]
?Sayan Sinha
09/25/2023, 10:14 PMAuto
models. It starts training and gets stuck, and never recovers. I am using Python 3.10.1 on Ubuntu 20.04.6 LTS.
In [1]: from neuralforecast import NeuralForecast
In [2]: from neuralforecast.auto import AutoNBEATS
In [3]: from neuralforecast.utils import AirPassengersDF
...:
...: Y_df = AirPassengersDF
...: Y_df.head()
Out[3]:
unique_id ds y
0 1.0 1949-01-31 112.0
1 1.0 1949-02-28 118.0
2 1.0 1949-03-31 132.0
3 1.0 1949-04-30 129.0
4 1.0 1949-05-31 121.0
In [4]: models = [AutoNBEATS(h=12)]
In [5]: nf = NeuralForecast(models=models, freq='M')
In [6]: nf.fit(df=Y_df)
2023-09-25 18:10:00,972 INFO worker.py:1621 -- Started a local Ray instance.
2023-09-25 18:10:01,764 INFO tune.py:226 -- Initializing Ray automatically. For cluster usage or custom Ray initialization, call `ray.init(...)` befor
e `Tuner(...)`.
2023-09-25 18:10:01,767 INFO tune.py:666 -- [output] This will use the new output engine with verbosity 0. To disable the new output and use the legac
y output engine, set the environment variable RAY_AIR_NEW_OUTPUT=0. For more information, please see <https://github.com/ray-project/ray/issues/36949>
+--------------------------------------------------------------------+
| Configuration for experiment _train_tune_2023-09-25_18-09-58 |
+--------------------------------------------------------------------+
| Search algorithm BasicVariantGenerator |
| Scheduler FIFOScheduler |
| Number of trials 10 |
+--------------------------------------------------------------------+
View detailed results here: <$USER>/ray_results/_train_tune_2023-09-25_18-09-58
To visualize your results with TensorBoard, run: `tensorboard --logdir <$USER>/ray_results/_train_tune_2023-09-25_18-09-58`
(_train_tune pid=401980) Global seed set to 10
(_train_tune pid=401980) [rank: 0] Global seed set to 10
(_train_tune pid=401980) Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/3
It does not progress beyond this point. Issuing $ ps aux | grep ray
points out to multiple ray::IDLE
processes. Any help regarding this would be great! Thanks.ロビン(Robin Daily Life)
09/26/2023, 7:02 AMShreya Mathur
09/26/2023, 4:23 PM