marah othman
08/01/2023, 8:47 AMMark
08/01/2023, 11:04 AMMark
08/01/2023, 11:12 AMMark
08/01/2023, 11:36 AMMark
08/01/2023, 3:47 PMMark
08/01/2023, 3:47 PMManuel
08/01/2023, 6:38 PMDAVID FERNANDO PARADA BRIJALBA
08/02/2023, 3:57 PMmarah othman
08/02/2023, 5:25 PMmarah othman
08/02/2023, 9:42 PMMark
08/03/2023, 8:07 PMPhil
08/03/2023, 8:43 PMsCRPS
with quantiles = [0.95] with a HINT class. My hope is that reconciling the distributions across the hierarchy might result in better estimates of the 95th quantiles.
2. Since the 95th quantiles themselves can't be reconciled. i.e. the sum of the 95th quantiles of the bottom nodes is not equal to the 95th quantile of the parent nodes. Am I better off getting rid of the HINT layer altogether?
3. Can anyone provide a heuristic guide to tuning NHITS? I want to get a feel for the hyper-parameter space before I use Ray to fine tune a model?Afiq Johari
08/07/2023, 4:24 PMverbose
is already False
but I would prefer to suppress it further.
def predict(
self,
df: Optional[pd.DataFrame] = None,
static_df: Optional[pd.DataFrame] = None,
futr_df: Optional[pd.DataFrame] = None,
sort_df: bool = True,
verbose: bool = False,
**data_kwargs,
):
Phil
08/07/2023, 7:52 PM---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
Cell In[32], line 1
----> 1 nf2 = NeuralForecast.load(path='./results/model5/')
File ~/pytorch_forecasting/silvertorch/pytorch/lib/python3.10/site-packages/neuralforecast/core.py:693, in NeuralForecast.load(path, verbose)
690 for model in models_ckpt:
691 model_name = model.split("_")[0]
692 models.append(
--> 693 MODEL_FILENAME_DICT[model_name].load_from_checkpoint(f"{path}/{model}")
694 )
695 if verbose:
696 print(f"Model {model_name} loaded.")
KeyError: 'hint'
marah othman
08/08/2023, 8:42 AMAntoine SCHWARTZ -CROIX-
08/08/2023, 9:40 AMinference_windows_batch_size
I should be able to handle it, but does that mean that for inference DeepAR doesn't keep only the last input_size
points as it does for training?
Thanks in advance!Phil
08/08/2023, 5:50 PMCyril de Catheu
08/09/2023, 10:44 PMtraining_step
, _create_windows
and _parse_windows
.
I’m a bit confused by the code.
Is there a paper or some doc that describes the idea of what’s done there?marah othman
08/11/2023, 12:34 PMStefan Wiegand
08/11/2023, 1:53 PMmarah othman
08/14/2023, 4:30 PMPhil
08/15/2023, 9:33 PMscaler_type
is behaving weirdly for the validation set loss functions. If it's not identity
. The validation loss is way off. What is going on?
I'm observing some really weird behavior out of the NBEATS model and I wanted to see if I am out to lunch or what. To get a sense of the model landscape. I started small. Out of my dataset of 100 timeseries, I wanted to find a basic set of hyper-parameters where I could overfit 10 timeseries really well. Once I felt comfortable with the complexity and relative space of model parameters, I would reduce the complexity, add regularization and try to get a model which could perform well on a validation set.
After some trial and error I found I could overfit my data very well with this model configuration. I plotted the training loss there.
model_name = "NBEATSx"
stacks = 2
params = dict(
h=180,
input_size=360,
loss=HuberLoss(),
max_steps=2000,
dropout_prob_theta=0.0,
stack_types=stacks*["identity"],
n_blocks=stacks*[15],
mlp_units=[[16, 16] for _ in range(stacks)],
scaler_type="standard",
learning_rate=1e-3,
random_seed=200,
alias=model_name,
batch_size=5,
)
nf = NeuralForecast(models=[NBEATSx(**params)], freq="D")
nf.fit(df=Y_train_df.reset_index())
Epoch 999: 100%
2/2 [00:00<00:00, 6.45it/s, v_num=788, train_loss_step=0.0128, train_loss_epoch=0.0122]
Next, I wanted to check the performance on a validation set. The main issue is no matter how I try to reduce the complexity. example: changing the number of stacks from 2 -> 1or number of blocks from 15 -> 1. or even number of mlp_units from 16 -> 2. Or adding dropout, The validation loss of MAPE or any other loss for that matter does not decrease significantly at all!!! See the second plot. The scale makes it look like it's decreasing but it gets to 1 - 1e-6.
stacks = 1
params = dict(
h=MODEL_HORIZON,
input_size=CONTEXT_LENGTH,
loss=HuberLoss(),
max_steps=2000,
dropout_prob_theta=0.5,
stack_types=stacks*["identity"],
n_blocks=stacks*[1],
mlp_units=[[16, 16] for _ in range(stacks)],
scaler_type="standard",
learning_rate=1e-3,
random_seed=200,
alias=model_name,
batch_size=5,
# Validation params
val_check_steps=1,
valid_loss=MAPE(),
early_stop_patience_steps=3000
)
However, the biggest change comes from changing the scaler_type
. If I change it to identity
. At least the MAPEs are in a decent range and I see some decrease and typical curve behavior.
What is going on?Manuel
08/16/2023, 7:52 AMLeonie
08/21/2023, 7:02 PMnf.fit(df_train, df_val)
Fan Tang
08/22/2023, 3:43 AMmarah othman
08/22/2023, 1:40 PMShreya Mathur
08/24/2023, 2:16 PMfrom ray.tune.search.basic_variant import BasicVariantGenerator
models = [AutoNHITS(h=61,
config=nhits_config,
loss=MAPE(),
search_alg=BasicVariantGenerator(random_state=42),
num_samples=100)]
nf = NeuralForecast(
models=models,
freq='D')
Y_hat_df = nf.cross_validation(df=Y_df, val_size=val_size,
test_size=test_size, n_windows=None)
When I run the same AutoNHITS model using neuralforecast version 1.6.2 (the latest version), the train loss step, train loss epoch for all configs turn to zero. The predictions are also zero. Did something change in the latest version?
I have attached the pics of the same. Thanks.Luis Huergo
08/24/2023, 7:00 PMLuis Huergo
08/24/2023, 8:49 PMPhil
08/24/2023, 9:27 PMnf2 = NeuralForecast.load(path='../cluster_results/results/b79f6fd0c50099a7519da800805bf436d4f4f4a6/')
RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.
I'm trying to debug this error on my own. I found this stackoverflow post. https://stackoverflow.com/questions/56369030/runtimeerror-attempting-to-deserialize-object-on-a-cuda-device
But I think this needs to be handle as part of the load function?https://github.com/Nixtla/neuralforecast/blob/1d78f503ac4cd0fdfa7aae9156a876fe94eb1db4/neuralforecast/core.py#L665
I think this can be solved if you have an extra argument to the load function map_location
MODEL_FILENAME_DICT[model_name].load_from_checkpoint(f"{path}/{model}", map_location=map_location)
If I specify map_location=torch.device('cpu'))
, it works