I m trying to log the metrics from AUTONHITS to our MLfLow t

Question

I m trying to log the metrics from AUTONHITS to our MLfLow tracking How can I set this up

José Morales · Answer

Hey Do you mean the results of each configuration

Phil · Answer

Yes that s right For each configuration I d like to have some kind of callback configured which could somewhow know the model id for that configuration and log the training and validation loss along with the chosen hyper parameters

Phil · Answer

I know there is a tensor board setup but It s difficult to access from our cluster If possible I d love to set it up so that it can log it in real time or atleast at every epoch or something like that

José Morales · Answer

Are you using ray or optuna

Phil · Answer

I m using ray

José Morales · Answer

Are you able to install from github

Phil · Answer

Depends on the package

José Morales · Answer

It d just be neuralforecast

José Morales · Answer

I just pushed some initial work to the callbacks branch it d be great if you could try it out It adds a `callbacks` argument to AutoNHITS where you can provide a list of callbacks like the one in the example <https docs ray io en latest tune tutorials tune metrics html|here>

José Morales · Answer

So if you can run `pip install git+<https github com nixtla neuralforecast git callbacks>` you should be able to use it

Phil · Answer

I won t be able to use it on our cluster since it automatically installs packages from our artifactory

Phil · Answer

but once this code is released I can add the latest release version to our package artifactory and try it then

Phil · Answer

Can I see your PR

José Morales · Answer

Sure it s <https github com Nixtla neuralforecast pull 795|this one> It needs more work because all the auto models need this argument I only added it to nhits

Phil · Answer

Very cool I guess LinkedIn is using it s own callback method called ```AIMLoggerCallback``` But I could add it to the callbacks arguments

José Morales · Answer

is it <https docs ray io en latest tune api doc ray tune logger aim AimLoggerCallback html|this one>

Phil · Answer

looks like it

Phil · Answer

I guess they tweaked it slighly since It is an internal repo but it looks the same

José Morales · Answer

ok this will be available in the next release

Phil · Answer

Thank you very much < José Morales>