hello, when I try to fit my model I get the follow...
# neural-forecastf
Francisco
10/20/2023, 3:05 PMhello, when I try to fit my model I get the following error:
Copy code
---------------------------------------------------------------------------
RaySystemError Traceback (most recent call last)
Cell In[18], line 1
----> 1 nf.fit(df=df_final)
File ~/anaconda3/envs/Merck2/lib/python3.10/site-packages/neuralforecast/core.py:274, in NeuralForecast.fit(self, df, static_df, val_size, sort_df, use_init_models, verbose)
271 print("WARNING: Deleting previously fitted models.")
273 for model in self.models:
--> 274 model.fit(self.dataset, val_size=val_size)
276 self._fitted = True
File ~/anaconda3/envs/Merck2/lib/python3.10/site-packages/neuralforecast/common/_base_auto.py:361, in BaseAuto.fit(self, dataset, val_size, test_size, random_seed)
359 val_size = val_size if val_size > 0 else self.h
360 if self.backend == "ray":
--> 361 results = self._tune_model(
362 cls_model=self.cls_model,
363 dataset=dataset,
364 val_size=val_size,
365 test_size=test_size,
366 cpus=self.cpus,
367 gpus=self.gpus,
368 verbose=self.verbose,
369 num_samples=self.num_samples,
370 search_alg=search_alg,
371 config=self.config,
372 )
373 best_config = results.get_best_result().config
374 else:
File ~/anaconda3/envs/Merck2/lib/python3.10/site-packages/neuralforecast/common/_base_auto.py:228, in BaseAuto._tune_model(self, cls_model, dataset, val_size, test_size, cpus, gpus, verbose, num_samples, search_alg, config)
215 def _tune_model(
216 self,
217 cls_model,
(...)
226 config,
227 ):
--> 228 train_fn_with_parameters = tune.with_parameters(
229 self._train_tune,
230 cls_model=cls_model,
231 dataset=dataset,
232 val_size=val_size,
233 test_size=test_size,
234 )
236 # Device
237 if gpus > 0:
File ~/anaconda3/envs/Merck2/lib/python3.10/site-packages/ray/tune/trainable/util.py:292, in with_parameters(trainable, **kwargs)
290 prefix = f"{str(trainable)}_"
291 for k, v in kwargs.items():
--> 292 parameter_registry.put(prefix + k, v)
294 trainable_name = getattr(trainable, "__name__", "tune_with_parameters")
295 keys = set(kwargs.keys())
File ~/anaconda3/envs/Merck2/lib/python3.10/site-packages/ray/tune/registry.py:296, in _ParameterRegistry.put(self, k, v)
294 self.to_flush[k] = v
295 if ray.is_initialized():
--> 296 self.flush()
File ~/anaconda3/envs/Merck2/lib/python3.10/site-packages/ray/tune/registry.py:308, in _ParameterRegistry.flush(self)
306 self.references[k] = v
307 else:
--> 308 self.references[k] = ray.put(v)
309 self.to_flush.clear()
File ~/anaconda3/envs/Merck2/lib/python3.10/site-packages/ray/_private/auto_init_hook.py:24, in wrap_auto_init.<locals>.auto_init_wrapper(*args, **kwargs)
21 @wraps(fn)
22 def auto_init_wrapper(*args, **kwargs):
23 auto_init_ray()
---> 24 return fn(*args, **kwargs)
File ~/anaconda3/envs/Merck2/lib/python3.10/site-packages/ray/_private/client_mode_hook.py:103, in client_mode_hook.<locals>.wrapper(*args, **kwargs)
101 if func.__name__ != "init" or is_client_mode_enabled_by_default:
102 return getattr(ray, func.__name__)(*args, **kwargs)
--> 103 return func(*args, **kwargs)
File ~/anaconda3/envs/Merck2/lib/python3.10/site-packages/ray/_private/worker.py:2620, in put(value, _owner)
2618 with profiling.profile("ray.put"):
2619 try:
-> 2620 object_ref = worker.put_object(value, owner_address=serialize_owner_address)
2621 except ObjectStoreFullError:
2622 <http://logger.info|logger.info>(
2623 "Put failed since the value was either too large or the "
2624 "store was full of pinned objects."
2625 )
File ~/anaconda3/envs/Merck2/lib/python3.10/site-packages/ray/_private/worker.py:707, in Worker.put_object(self, value, object_ref, owner_address)
699 raise TypeError(msg) from e
700 # This *must* be the first place that we construct this python
701 # ObjectRef because an entry with 0 local references is created when
702 # the object is Put() in the core worker, expecting that this python
703 # reference will be created. If another reference is created and
704 # removed before this one, it will corrupt the state in the
705 # reference counter.
706 return ray.ObjectRef(
--> 707 self.core_worker.put_serialized_object_and_increment_local_ref(
708 serialized_value, object_ref=object_ref, owner_address=owner_address
709 ),
710 # The initial local reference is already acquired internally.
711 skip_adding_local_ref=True,
712 )
File python/ray/_raylet.pyx:3260, in ray._raylet.CoreWorker.put_serialized_object_and_increment_local_ref()
File python/ray/_raylet.pyx:3152, in ray._raylet.CoreWorker._create_put_buffer()
File python/ray/_raylet.pyx:457, in ray._raylet.check_status()
RaySystemError: System error: Broken pipem
Max (Nixtla)
10/20/2023, 5:17 PMHi! Could you try the optuna backend instead?
f
Francisco
10/20/2023, 7:28 PMyes I tried but now I am getting this error
Francisco
10/20/2023, 7:30 PM Copy code
File "/Users/franciscotafur/anaconda3/envs/Merck2/lib/python3.10/site-packages/neuralforecast/models/nhits.py", line 43, in forward
forecast = F.interpolate(
File "/Users/franciscotafur/anaconda3/envs/Merck2/lib/python3.10/site-packages/torch/nn/functional.py", line 3954, in interpolate
return torch._C._nn.upsample_linear1d(input, output_size, align_corners, scale_factors)
NotImplementedError: The operator 'aten::upsample_linear1d.out' is not currently implemented for the MPS device. If you want this op to be added in priority during the prototype phase of this feature, please comment on <https://github.com/pytorch/pytorch/issues/77764>. As a temporary fix, you can set the environment variable `PYTORCH_ENABLE_MPS_FALLBACK=1` to use the CPU as a fallback for this op. WARNING: this will be slower than running natively on MPS.
[W 2023-10-20 14:23:25,600] Trial 0 failed with value None.
---------------------------------------------------------------------------
NotImplementedError Traceback (most recent call last)
Cell In[45], line 1
----> 1 nf.fit(df=df_final)
File ~/anaconda3/envs/Merck2/lib/python3.10/site-packages/neuralforecast/core.py:274, in NeuralForecast.fit(self, df, static_df, val_size, sort_df, use_init_models, verbose)
271 print("WARNING: Deleting previously fitted models.")
273 for model in self.models:
--> 274 model.fit(self.dataset, val_size=val_size)
276 self._fitted = True
File ~/anaconda3/envs/Merck2/lib/python3.10/site-packages/neuralforecast/common/_base_auto.py:375, in BaseAuto.fit(self, dataset, val_size, test_size, random_seed)
373 best_config = results.get_best_result().config
374 else:
--> 375 results = self._optuna_tune_model(
376 cls_model=self.cls_model,
377 dataset=dataset,
378 val_size=val_size,
379 test_size=test_size,
380 verbose=self.verbose,
381 num_samples=self.num_samples,
382 search_alg=search_alg,
383 config=self.config,
384 )
385 best_config = results.best_trial.user_attrs["ALL_PARAMS"]
386 self.model = self._fit_model(
387 cls_model=self.cls_model,
388 config=best_config,
(...)
391 test_size=test_size,
392 )
File ~/anaconda3/envs/Merck2/lib/python3.10/site-packages/neuralforecast/common/_base_auto.py:327, in BaseAuto._optuna_tune_model(self, cls_model, dataset, val_size, test_size, verbose, num_samples, search_alg, config)
324 sampler = None
326 study = optuna.create_study(sampler=sampler, direction="minimize")
--> 327 study.optimize(
328 objective,
329 n_trials=num_samples,
330 show_progress_bar=verbose,
331 )
332 return study
File ~/anaconda3/envs/Merck2/lib/python3.10/site-packages/optuna/study/study.py:442, in Study.optimize(self, func, n_trials, timeout, n_jobs, catch, callbacks, gc_after_trial, show_progress_bar)
339 def optimize(
340 self,
341 func: ObjectiveFuncType,
(...)
348 show_progress_bar: bool = False,
349 ) -> None:
350 """Optimize an objective function.
351
352 Optimization is done by choosing a suitable set of hyperparameter values from a given
(...)
440 If nested invocation of this method occurs.
441 """
--> 442 _optimize(
443 study=self,
444 func=func,
445 n_trials=n_trials,
446 timeout=timeout,
447 n_jobs=n_jobs,
448 catch=tuple(catch) if isinstance(catch, Iterable) else (catch,),
449 callbacks=callbacks,
450 gc_after_trial=gc_after_trial,
451 show_progress_bar=show_progress_bar,
452 )
File ~/anaconda3/envs/Merck2/lib/python3.10/site-packages/optuna/study/_optimize.py:66, in _optimize(study, func, n_trials, timeout, n_jobs, catch, callbacks, gc_after_trial, show_progress_bar)
64 try:
65 if n_jobs == 1:
---> 66 _optimize_sequential(
67 study,
68 func,
69 n_trials,
70 timeout,
71 catch,
72 callbacks,
73 gc_after_trial,
74 reseed_sampler_rng=False,
75 time_start=None,
76 progress_bar=progress_bar,
77 )
78 else:
79 if n_jobs == -1:
File ~/anaconda3/envs/Merck2/lib/python3.10/site-packages/optuna/study/_optimize.py:163, in _optimize_sequential(study, func, n_trials, timeout, catch, callbacks, gc_after_trial, reseed_sampler_rng, time_start, progress_bar)
160 break
162 try:
--> 163 frozen_trial = _run_trial(study, func, catch)
164 finally:
165 # The following line mitigates memory problems that can be occurred in some
166 # environments (e.g., services that use computing containers such as GitHub Actions).
167 # Please refer to the following PR for further details:
168 # <https://github.com/optuna/optuna/pull/325>.
169 if gc_after_trial:
File ~/anaconda3/envs/Merck2/lib/python3.10/site-packages/optuna/study/_optimize.py:251, in _run_trial(study, func, catch)
244 assert False, "Should not reach."
246 if (
247 frozen_trial.state == TrialState.FAIL
248 and func_err is not None
249 and not isinstance(func_err, catch)
250 ):
--> 251 raise func_err
252 return frozen_trial
File ~/anaconda3/envs/Merck2/lib/python3.10/site-packages/optuna/study/_optimize.py:200, in _run_trial(study, func, catch)
198 with get_heartbeat_thread(trial._trial_id, study._storage):
199 try:
--> 200 value_or_values = func(trial)
201 except exceptions.TrialPruned as e:
202 # TODO(mamu): Handle multi-objective cases.
203 state = TrialState.PRUNED
File ~/anaconda3/envs/Merck2/lib/python3.10/site-packages/neuralforecast/common/_base_auto.py:311, in BaseAuto._optuna_tune_model.<locals>.objective(trial)
309 def objective(trial):
310 cfg = config(trial)
--> 311 fitted_model = self._fit_model(
312 cls_model=cls_model,
313 config=cfg,
314 dataset=dataset,
315 val_size=val_size,
316 test_size=test_size,
317 )
318 trial.set_user_attr("ALL_PARAMS", cfg)
319 return fitted_model.trainer.callback_metrics["valid_loss"].item()
File ~/anaconda3/envs/Merck2/lib/python3.10/site-packages/neuralforecast/common/_base_auto.py:336, in BaseAuto._fit_model(self, cls_model, config, dataset, val_size, test_size)
334 def _fit_model(self, cls_model, config, dataset, val_size, test_size):
335 model = cls_model(**config)
--> 336 model.fit(dataset, val_size=val_size, test_size=test_size)
337 return model
FTrainer._run_stage(self)
1032 if self.training:
1033 with isolate_rng():
-> 1034 self._run_sanity_check()
1035 with torc
387 self._build_step_args_from_hook_kwargs(hook_kwargs, hook_name)
567 insample_mask=insample_mask, # [Ws, L]
(...)
570 stat_exog=stat_exog,
571 ) # [Ws, 1]
573 # Model Predictions
--> 574 output_batch = self(windows_batch)
575 valid_loss_batch = self._compute_valid_loss(
576 outsample_y=original_outsample_y,
577 output=output_batch,
578 outsample_mask=outsample_mask,
579 temporal_cols=batch["temporal_cols"],
580 )
581 valid_losses.append(valid_loss_batch)
File ~/anaconda3/envs/Merck2/lib/python3.10/site-packages/torch/nn/modules/module.py:1501, in Module._call_impl(self, *args, **kwargs)
1496 # If we don't have any hooks, we want to skip the rest of the logic in
1497 # this function, and just call forward.
1498 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
1499 or _global_backward_pre_hooks or _global_backward_hooks
1500 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1501 return forward_call(*args, **kwargs)
1502 # Do not call functions when jit is used
1503 full_backward_hooks, non_full_backward_hooks = [], []
File ~/anaconda3/envs/Merck2/lib/python3.10/site-packages/neuralforecast/models/nhits.py:392, in NHITS.forward(self, windows_batch)
390 block_forecasts = [forecast.repeat(1, self.h, 1)]
391 for i, block in enumerate(self.blocks):
--> 392 backcast, block_forecast = block(
393 insample_y=residuals,
394 futr_exog=futr_exog,
395 hist_exog=hist_exog,
396 stat_exog=stat_exog,
397 )
398 residuals = (residuals - backcast) * insample_mask
399 forecast = forecast + block_forecast
File ~/anaconda3/envs/Merck2/lib/python3.10/site-packages/torch/nn/modules/module.py:1501, in Module._call_impl(self, *args, **kwargs)
1496 # If we don't have any hooks, we want to skip the rest of the logic in
1497 # this function, and just call forward.
1498 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
1499 or _global_backward_pre_hooks or _global_backward_hooks
1500 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1501 return forward_call(*args, **kwargs)
1502 # Do not call functions when jit is used
1503 full_backward_hooks, non_full_backward_hooks = [], []
File ~/anaconda3/envs/Merck2/lib/python3.10/site-packages/neuralforecast/models/nhits.py:179, in NHITSBlock.forward(self, insample_y, futr_exog, hist_exog, stat_exog)
177 # Compute local projection weights and projection
178 theta = self.layers(insample_y)
--> 179 backcast, forecast = self.basis(theta)
180 return backcast, forecast
File ~/anaconda3/envs/Merck2/lib/python3.10/site-packages/torch/nn/modules/module.py:1501, in Module._call_impl(self, *args, **kwargs)
1496 # If we don't have any hooks, we want to skip the rest of the logic in
1497 # this function, and just call forward.
1498 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
1499 or _global_backward_pre_hooks or _global_backward_hooks
1500 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1501 return forward_call(*args, **kwargs)
1502 # Do not call functions when jit is used
1503 full_backward_hooks, non_full_backward_hooks = [], []
File ~/anaconda3/envs/Merck2/lib/python3.10/site-packages/neuralforecast/models/nhits.py:43, in _IdentityBasis.forward(self, theta)
40 knots = knots.reshape(len(knots), self.out_features, -1)
41 if self.interpolation_mode in ["nearest", "linear"]:
42 # knots = knots[:,None,:]
---> 43 forecast = F.interpolate(
44 knots, size=self.forecast_size, mode=self.interpolation_mode
45 )
46 # forecast = forecast[:,0,:]
47 elif "cubic" in self.interpolation_mode:
File ~/anaconda3/envs/Merck2/lib/python3.10/site-packages/torch/nn/functional.py:3954, in interpolate(input, size, scale_factor, mode, align_corners, recompute_scale_factor, antialias)
3952 if input.dim() == 3 and mode == "linear":
3953 assert align_corners is not None
-> 3954 return torch._C._nn.upsample_linear1d(input, output_size, align_corners, scale_factors)
3955 if input.dim() == 4 and mode == "bilinear":
3956 assert align_corners is not None
NotImplementedError: The operator 'aten::upsample_linear1d.out' is not currently implemented for the MPS device. If you want this op to be added in priority during the prototype phase of this feature, please comment on <https://github.com/pytorch/pytorch/issues/77764>. As a temporary fix, you can set the environment variable `PYTORCH_ENABLE_MPS_FALLBACK=1` to use the CPU as a fallback for this op. WARNING: this will be slower than running natively on MPS.Francisco
10/20/2023, 7:30 PMI tried setting the environment variable but it still isn't working
m
Max (Nixtla)
10/20/2023, 7:45 PM@José Morales, any idea?
j
José Morales
10/20/2023, 7:47 PMHow did you set the env variable?
f
Francisco
10/20/2023, 7:49 PMlike this import os
os.environ['PYTORCH_ENABLE_MPS_FALLBACK'] = '1'
Francisco
10/20/2023, 7:49 PMI also tried this:
torch.device("cpu")
j
José Morales
10/20/2023, 8:11 PMThis may need to be set at startup. Did you restart your kernel?
José Morales
10/20/2023, 8:11 PMIt's probably best if you set it before your other imports
f
Francisco
10/20/2023, 8:15 PMIt worked
Francisco
10/20/2023, 8:15 PMthank you