Question from a user. Maybe @José Morales @Marco @Yibei you have recommendations? Thank you!

Does it slow down inference if someone uses fine-tuning? If so, how much slower? What's best practice if they need fast inference for multiple calls in bursts?

Yes, it slows down a bit. How much slower, it will depend on the number of

finetune_steps

. The more steps we finetune, the longer it takes, but it can also lead to overfitting, so we don't want to finetune too much either. My advice would be to test different finetuning steps, see how it impacts speed and performance. Not sure how fast they want inference to be, but even with finetuning, it will be relatively faster than training a model from scratch.

Great, thank you! Shared that with the user.