hello when I try to fit my model I get the following error `

Question

hello when I try to fit my model I get the following error ``` RaySystemError Traceback most recent call last Cell In 18 line 1 gt 1 nf fit df=df final File ~ anaconda3 envs Merck2 lib python3 10 site packages neuralforecast core py 274 in NeuralForecast fit self df static df val size sort df use init models verbose 271 print WARNING Deleting previously fitted models 273 for model in self models gt 274 model fit self dataset val size=val size 276 self fitted = True File ~ anaconda3 envs Merck2 lib python3 10 site packages neuralforecast common base auto py 361 in BaseAuto fit self dataset val size test size random seed 359 val size = val size if val size gt 0 else self h 360 if self backend == ray gt 361 results = self tune model 362 cls model=self cls model 363 dataset=dataset 364 val size=val size 365 test size=test size 366 cpus=self cpus 367 gpus=self gpus 368 verbose=self verbose 369 num samples=self num samples 370 search alg=search alg 371 config=self config 372 373 best config = results get best result config 374 else File ~ anaconda3 envs Merck2 lib python3 10 site packages neuralforecast common base auto py 228 in BaseAuto tune model self cls model dataset val size test size cpus gpus verbose num samples search alg config 215 def tune model 216 self 217 cls model 226 config 227 gt 228 train fn with parameters = tune with parameters 229 self train tune 230 cls model=cls model 231 dataset=dataset 232 val size=val size 233 test size=test size 234 236 Device 237 if gpus gt 0 File ~ anaconda3 envs Merck2 lib python3 10 site packages ray tune trainable util py 292 in with parameters trainable kwargs 290 prefix = f str trainable 291 for k v in kwargs items gt 292 parameter registry put prefix + k v 294 trainable name = getattr trainable name tune with parameters 295 keys = set kwargs keys File ~ anaconda3 envs Merck2 lib python3 10 site packages ray tune registry py 296 in ParameterRegistry put self k v 294 self to flush k = v 295 if ray is initialized gt 296 self flush File ~ anaconda3 envs Merck2 lib python3 10 site packages ray tune registry py 308 in ParameterRegistry flush self 306 self references k = v 307 else gt 308 self references k = ray put v 309 self to flush clear File ~ anaconda3 envs Merck2 lib python3 10 site packages ray private auto init hook py 24 in wrap auto init lt locals gt auto init wrapper args kwargs 21 wraps fn 22 def auto init wrapper args kwargs 23 auto init ray gt 24 return fn args kwargs File ~ anaconda3 envs Merck2 lib python3 10 site packages ray private client mode hook py 103 in client mode hook lt locals gt wrapper args kwargs 101 if func name = init or is client mode enabled by default 102 return getattr ray func name args kwargs gt 103 return func args kwargs File ~ anaconda3 envs Merck2 lib python3 10 site packages ray private worker py 2620 in put value owner 2618 with profiling profile ray put 2619 try gt 2620 object ref = worker put object value owner address=serialize owner address 2621 except ObjectStoreFullError 2622 <http logger info|logger info> 2623 Put failed since the value was either too large or the 2624 store was full of pinned objects 2625 File ~ anaconda3 envs Merck2 lib python3 10 site packages ray private worker py 707 in Worker put object self value object ref owner address 699 raise TypeError msg from e 700 This must be the first place that we construct this python 701 ObjectRef because an entry with 0 local references is created when 702 the object is Put in the core worker expecting that this python 703 reference will be created If another reference is created and 704 removed before this one it will corrupt the state in the 705 reference counter 706 return ray ObjectRef gt 707 self core worker put serialized object and increment local ref 708 serialized value object ref=object ref owner address=owner address 709 710 The initial local reference is already acquired internally 711 skip adding local ref=True 712 File python ray raylet pyx 3260 in ray raylet CoreWorker put serialized object and increment local ref File python ray raylet pyx 3152 in ray raylet CoreWorker create put buffer File python ray raylet pyx 457 in ray raylet check status RaySystemError System error Broken pipe```

Max (Nixtla) · Answer

Hi Could you try the <https nixtla github io neuralforecast examples automatic hyperparameter tuning html optuna backend|optuna backend> instead

Francisco · Answer

yes I tried but now I am getting this error

Francisco · Answer

``` File Users franciscotafur anaconda3 envs Merck2 lib python3 10 site packages neuralforecast models nhits py line 43 in forward forecast = F interpolate File Users franciscotafur anaconda3 envs Merck2 lib python3 10 site packages torch nn functional py line 3954 in interpolate return torch C nn upsample linear1d input output size align corners scale factors NotImplementedError The operator aten upsample linear1d out is not currently implemented for the MPS device If you want this op to be added in priority during the prototype phase of this feature please comment on As a temporary fix you can set the environment variable `PYTORCH ENABLE MPS FALLBACK=1` to use the CPU as a fallback for this op WARNING this will be slower than running natively on MPS W 2023 10 20 14 23 25 600 Trial 0 failed with value None NotImplementedError Traceback most recent call last Cell In 45 line 1 gt 1 nf fit df=df final File ~ anaconda3 envs Merck2 lib python3 10 site packages neuralforecast core py 274 in NeuralForecast fit self df static df val size sort df use init models verbose 271 print WARNING Deleting previously fitted models 273 for model in self models gt 274 model fit self dataset val size=val size 276 self fitted = True File ~ anaconda3 envs Merck2 lib python3 10 site packages neuralforecast common base auto py 375 in BaseAuto fit self dataset val size test size random seed 373 best config = results get best result config 374 else gt 375 results = self optuna tune model 376 cls model=self cls model 377 dataset=dataset 378 val size=val size 379 test size=test size 380 verbose=self verbose 381 num samples=self num samples 382 search alg=search alg 383 config=self config 384 385 best config = results best trial user attrs ALL PARAMS 386 self model = self fit model 387 cls model=self cls model 388 config=best config 391 test size=test size 392 File ~ anaconda3 envs Merck2 lib python3 10 site packages neuralforecast common base auto py 327 in BaseAuto optuna tune model self cls model dataset val size test size verbose num samples search alg config 324 sampler = None 326 study = optuna create study sampler=sampler direction= minimize gt 327 study optimize 328 objective 329 n trials=num samples 330 show progress bar=verbose 331 332 return study File ~ anaconda3 envs Merck2 lib python3 10 site packages optuna study study py 442 in Study optimize self func n trials timeout n jobs catch callbacks gc after trial show progress bar 339 def optimize 340 self 341 func ObjectiveFuncType 348 show progress bar bool = False 349 gt None 350 Optimize an objective function 351 352 Optimization is done by choosing a suitable set of hyperparameter values from a given 440 If nested invocation of this method occurs 441 gt 442 optimize 443 study=self 444 func=func 445 n trials=n trials 446 timeout=timeout 447 n jobs=n jobs 448 catch=tuple catch if isinstance catch Iterable else catch 449 callbacks=callbacks 450 gc after trial=gc after trial 451 show progress bar=show progress bar 452 File ~ anaconda3 envs Merck2 lib python3 10 site packages optuna study optimize py 66 in optimize study func n trials timeout n jobs catch callbacks gc after trial show progress bar 64 try 65 if n jobs == 1 gt 66 optimize sequential 67 study 68 func 69 n trials 70 timeout 71 catch 72 callbacks 73 gc after trial 74 reseed sampler rng=False 75 time start=None 76 progress bar=progress bar 77 78 else 79 if n jobs == 1 File ~ anaconda3 envs Merck2 lib python3 10 site packages optuna study optimize py 163 in optimize sequential study func n trials timeout catch callbacks gc after trial reseed sampler rng time start progress bar 160 break 162 try gt 163 frozen trial = run trial study func catch 164 finally 165 The following line mitigates memory problems that can be occurred in some 166 environments e g services that use computing containers such as GitHub Actions 167 Please refer to the following PR for further details 168 169 if gc after trial File ~ anaconda3 envs Merck2 lib python3 10 site packages optuna study optimize py 251 in run trial study func catch 244 assert False Should not reach 246 if 247 frozen trial state == TrialState FAIL 248 and func err is not None 249 and not isinstance func err catch 250 gt 251 raise func err 252 return frozen trial File ~ anaconda3 envs Merck2 lib python3 10 site packages optuna study optimize py 200 in run trial study func catch 198 with get heartbeat thread trial trial id study storage 199 try gt 200 value or values = func trial 201 except exceptions TrialPruned as e 202 TODO mamu Handle multi objective cases 203 state = TrialState PRUNED File ~ anaconda3 envs Merck2 lib python3 10 site packages neuralforecast common base auto py 311 in BaseAuto optuna tune model lt locals gt objective trial 309 def objective trial 310 cfg = config trial gt 311 fitted model = self fit model 312 cls model=cls model 313 config=cfg 314 dataset=dataset 315 val size=val size 316 test size=test size 317 318 trial set user attr ALL PARAMS cfg 319 return fitted model trainer callback metrics valid loss item File ~ anaconda3 envs Merck2 lib python3 10 site packages neuralforecast common base auto py 336 in BaseAuto fit model self cls model config dataset val size test size 334 def fit model self cls model config dataset val size test size 335 model = cls model config gt 336 model fit dataset val size=val size test size=test size 337 return model FTrainer run stage self 1032 if self training 1033 with isolate rng gt 1034 self run sanity check 1035 with torc 387 self build step args from hook kwargs hook kwargs hook name 567 insample mask=insample mask Ws L 570 stat exog=stat exog 571 Ws 1 573 Model Predictions gt 574 output batch = self windows batch 575 valid loss batch = self compute valid loss 576 outsample y=original outsample y 577 output=output batch 578 outsample mask=outsample mask 579 temporal cols=batch temporal cols 580 581 valid losses append valid loss batch File ~ anaconda3 envs Merck2 lib python3 10 site packages torch nn modules module py 1501 in Module call impl self args kwargs 1496 If we don t have any hooks we want to skip the rest of the logic in 1497 this function and just call forward 1498 if not self backward hooks or self backward pre hooks or self forward hooks or self forward pre hooks 1499 or global backward pre hooks or global backward hooks 1500 or global forward hooks or global forward pre hooks gt 1501 return forward call args kwargs 1502 Do not call functions when jit is used 1503 full backward hooks non full backward hooks = File ~ anaconda3 envs Merck2 lib python3 10 site packages neuralforecast models nhits py 392 in NHITS forward self windows batch 390 block forecasts = forecast repeat 1 self h 1 391 for i block in enumerate self blocks gt 392 backcast block forecast = block 393 insample y=residuals 394 futr exog=futr exog 395 hist exog=hist exog 396 stat exog=stat exog 397 398 residuals = residuals backcast insample mask 399 forecast = forecast + block forecast File ~ anaconda3 envs Merck2 lib python3 10 site packages torch nn modules module py 1501 in Module call impl self args kwargs 1496 If we don t have any hooks we want to skip the rest of the logic in 1497 this function and just call forward 1498 if not self backward hooks or self backward pre hooks or self forward hooks or self forward pre hooks 1499 or global backward pre hooks or global backward hooks 1500 or global forward hooks or global forward pre hooks gt 1501 return forward call args kwargs 1502 Do not call functions when jit is used 1503 full backward hooks non full backward hooks = File ~ anaconda3 envs Merck2 lib python3 10 site packages neuralforecast models nhits py 179 in NHITSBlock forward self insample y futr exog hist exog stat exog 177 Compute local projection weights and projection 178 theta = self layers insample y gt 179 backcast forecast = self basis theta 180 return backcast forecast File ~ anaconda3 envs Merck2 lib python3 10 site packages torch nn modules module py 1501 in Module call impl self args kwargs 1496 If we don t have any hooks we want to skip the rest of the logic in 1497 this function and just call forward 1498 if not self backward hooks or self backward pre hooks or self forward hooks or self forward pre hooks 1499 or global backward pre hooks or global backward hooks 1500 or global forward hooks or global forward pre hooks gt 1501 return forward call args kwargs 1502 Do not call functions when jit is used 1503 full backward hooks non full backward hooks = File ~ anaconda3 envs Merck2 lib python3 10 site packages neuralforecast models nhits py 43 in IdentityBasis forward self theta 40 knots = knots reshape len knots self out features 1 41 if self interpolation mode in nearest linear 42 knots = knots None gt 43 forecast = F interpolate 44 knots size=self forecast size mode=self interpolation mode 45 46 forecast = forecast 0 47 elif cubic in self interpolation mode File ~ anaconda3 envs Merck2 lib python3 10 site packages torch nn functional py 3954 in interpolate input size scale factor mode align corners recompute scale factor antialias 3952 if input dim == 3 and mode == linear 3953 assert align corners is not None gt 3954 return torch C nn upsample linear1d input output size align corners scale factors 3955 if input dim == 4 and mode == bilinear 3956 assert align corners is not None NotImplementedError The operator aten upsample linear1d out is not currently implemented for the MPS device If you want this op to be added in priority during the prototype phase of this feature please comment on As a temporary fix you can set the environment variable `PYTORCH ENABLE MPS FALLBACK=1` to use the CPU as a fallback for this op WARNING this will be slower than running natively on MPS ```

Francisco · Answer

I tried setting the environment variable but it still isn t working

Max (Nixtla) · Answer

< José Morales> any idea

José Morales · Answer

How did you set the env variable

Francisco · Answer

like this import os os environ PYTORCH ENABLE MPS FALLBACK = 1

Francisco · Answer

I also tried this torch device cpu

José Morales · Answer

This may need to be set at startup Did you restart your kernel

José Morales · Answer

It s probably best if you set it before your other imports

Francisco · Answer

It worked

Francisco · Answer

thank you