I'm trying to log the metrics from AUTONHITS to ou...
# neural-forecastp
Phil
10/23/2023, 9:17 PMI'm trying to log the metrics from AUTONHITS to our MLfLow tracking. How can I set this up?
j
José Morales
10/23/2023, 9:26 PMHey. Do you mean the results of each configuration?
p
Phil
10/23/2023, 9:27 PMYes that's right. For each configuration, I'd like to have some kind of callback configured which could somewhow know the model id for that configuration and log the training and validation loss along with the chosen hyper-parameters
Phil
10/23/2023, 9:28 PMI know there is a tensor board setup but It's difficult to access from our cluster... If possible, I'd love to set it up so that it can log it in real time ( or atleast at every epoch) or something like that
j
José Morales
10/23/2023, 9:30 PMAre you using ray or optuna?
p
Phil
10/23/2023, 9:30 PMI'm using ray
j
José Morales
10/23/2023, 10:20 PMAre you able to install from github?
p
Phil
10/23/2023, 10:21 PMDepends on the package?
j
José Morales
10/23/2023, 10:21 PMIt'd just be neuralforecast
José Morales
10/23/2023, 10:23 PMI just pushed some initial work to the callbacks branch, it'd be great if you could try it out. It adds a
callbacksJosé Morales
10/23/2023, 10:23 PMSo if you can run:
pip install git+<https://github.com/nixtla/neuralforecast.git@callbacks>p
Phil
10/23/2023, 10:24 PMI won't be able to use it on our cluster since it automatically installs packages from our artifactory
Phil
10/23/2023, 10:24 PMbut once this code is released, I can add the latest release version to our package artifactory and try it then
Phil
10/23/2023, 10:25 PMCan I see your PR?
j
José Morales
10/23/2023, 10:27 PMSure, it's this one. It needs more work because all the auto models need this argument, I only added it to nhits
p
Phil
10/23/2023, 10:28 PMVery cool. I guess LinkedIn is using it's own callback method called:
Copy code
AIMLoggerCallbackj
José Morales
10/23/2023, 10:30 PMis it this one?
p
Phil
10/23/2023, 10:31 PMlooks like it
Phil
10/23/2023, 10:31 PMI guess they tweaked it slighly since It is an internal repo but it looks the same
j
José Morales
10/23/2023, 10:51 PMok. this will be available in the next release
❤️ 1
p
Phil
10/23/2023, 10:51 PMThank you very much @José Morales!