Another one for you <@U06JPAURJBE> about the resul...
# supportt
Tracy Teal
07/01/2024, 10:26 PMAnother one for you @Yibei about the results from a deterministic simulated dataset. Message in thread. Thank you!
✅ 1
Tracy Teal
07/01/2024, 10:27 PMZakrisson, Henning <Zakrisson.Henning@bcg.com>
Hi! I am a data scientist looking into using transformer-based time series models and am currently looking into TimeGPT.
As a small test I tried using it to forecast a simple SARIMA model which is virtually deterministic. I tried both with and without fine-tuning (100 steps) and as a benchmark I also tried simply passing the data to ChatGPT and asking it to complete the sequence. These are my results:
Which to me looks surprisingly bad and made me wonder if I am doing something wrong. The time series should be incredibly easy to forecast, and is very similar to the airline passenger set you include as an example in your quick start guide.
I attached a simple python script to reproduce my results – can you spot any issues with how I am setting this up?
Tracy Teal
07/01/2024, 10:27 PMimage.png
Tracy Teal
07/01/2024, 10:27 PMsarima_test copy.txt
y
Yibei
07/01/2024, 10:46 PMHi Tracy! I think the poor performance is due to the overly long forecast horizon. Usually, TimeGPT perform well when forecasting horizon is predict less than one seasonal cycle. In this case, I think he should try to use a shorter horizon
hmodel=timegpt-1-long-horizon🙏 1
✅ 1
t
Tracy Teal
07/02/2024, 11:18 PMHi @Yibei, it looks like he's not doing long term forecasting. Do you have other ideas on why he might not be seeing very accurate forecasts?
I understand that, but as you can see in the image, the accuracy is quite poor even for short horizons and this does not change when I adjust the horizon to only predict values that are closer into the future. Also, the horizon is not even as long as the one in your quickstart example, which is a very, very, similar problem to this.
So, just to be clear: the accuracy I am seeing is expected?
y
Yibei
07/02/2024, 11:19 PMLet me run his code and get back:)
🙏 1
Yibei
07/03/2024, 12:27 AMI decreased the horizon and increased the finetune steps to achieve this. The time series provided is short and deterministic. Increase the finetuning step should help to better capture the behavior:)
t
Tracy Teal
07/03/2024, 12:33 AMthank you! And for including the output image too.
🫶 1