nixtlacommunity #general

Channels

random

linen

statsforecast

Phi Nguyen

09/08/2022, 4:39 PM

image.png

Phi Nguyen

09/08/2022, 4:42 PM

let me share a reproducible script

Phi Nguyen

09/08/2022, 4:43 PM

Copy code

from time import time
import ray
import pandas as pd
#from neuralforecast.data.datasets.m5 import M5, M5Evaluation
from statsforecast import StatsForecast
from statsforecast.models import ETS

ray.init(address="auto")

Y_df = pd.read_parquet('<s3://m5-benchmarks/data/train/target.parquet>')
Y_df = Y_df.rename(columns={
    'item_id': 'unique_id', 
    'timestamp': 'ds', 
    'demand': 'y'
})
Y_df['ds'] = pd.to_datetime(Y_df['ds'])

Y_df = Y_df[Y_df.unique_id == "FOODS_1_001_CA_1"]

constant = 10
Y_df['y'] += constant

fcst = StatsForecast(
    df=Y_df, 
    models=[ETS(season_length=7, model='ZNA')], 
    freq='D'
)

Y_hat = fcst.forecast(28)

Max (Nixtla)

09/08/2022, 4:53 PM

Y_df = Y_df[Y_df.unique_id == "FOODS_1_001_CA_1"]

# The problem was that the type was a category. # Add this line

Y_df['unique_id'] = Y_df['unique_id'].astype(str)

Max (Nixtla)

09/08/2022, 4:54 PM

Copy code

from time import time
import ray
import pandas as pd
#from neuralforecast.data.datasets.m5 import M5, M5Evaluation
from statsforecast import StatsForecast
from statsforecast.models import ETS

ray.init(address="auto")

Y_df = pd.read_parquet('<s3://m5-benchmarks/data/train/target.parquet>')
Y_df = Y_df.rename(columns={
    'item_id': 'unique_id', 
    'timestamp': 'ds', 
    'demand': 'y'
})
Y_df['ds'] = pd.to_datetime(Y_df['ds'])

Y_df = Y_df[Y_df.unique_id == "FOODS_1_001_CA_1"]
# Add this line
Y_df['unique_id'] = Y_df['unique_id'].astype(str)

constant = 10
Y_df['y'] += constant

fcst = StatsForecast(
    df=Y_df, 
    models=[ETS(season_length=7, model='ZNA')], 
    freq='D'
)

Y_hat = fcst.forecast(28)

Phi Nguyen

09/08/2022, 5:35 PM

Thanks! it worked!

🎉 1

Tia Guo

09/12/2022, 4:09 PM

Hi, I am forecasting weekly data, meaning seasonal_length = 52. Meanwhile, I am getting the error message as NameError: name 'warnings' is not defined. The detailed message shows 'warnings.warn("I can't handle data with frequency greater than 24. ""Seasonality will be ignored."' I also tried to simply remove the season_length argument. It won't give me error, but I got consistant forecast result. I am wondering how can I run forecasting with greater than 24 frequency while not getting this error message. thanks!

👀 1

Jonathan Farland

09/13/2022, 9:47 PM

Hi everyone - I work at H2O.ai and have really enjoyed exploring Nixtla's tools. A lot of my focus in the last decade has been on building large-scale, high-frequency forecasting systems, so I love to see some of the greatest R packages getting some Pythonic love! I'm looking into using

HierarchicalForecast

for a few applications - are there are utility or helper functions to generate the

matrix for a given hierarchy? thanks in advance

👀 1

Ginger Holt

09/14/2022, 4:50 PM

I am using hierarchicalforecast to reconcile entity-level forecasts. Does anyone have experience in dealing with new entities entering the hierarchy? Basically, the hierarchy is not constant over time. The specific example is forecasting revenue at these levels: account, sub-business unit, and business unit for 1 quarter ahead. New accounts will enter the hierarchy, and I want to account for that in my higher-level (sub-business unit, business unit) forecasts.

👀 1

Emre Varol

09/19/2022, 8:32 AM

Hi everyone, I enjoy using Nixtla libraries particularly

mlforecast

. They are quite speedy. On the other hand, after upgrading the

xgboost

Colab

, I realized that the forecasts have changed dramatically. Could you please check the pic and the notebook below? Any suggestions will be welcomed. @Max (Nixtla) @fede (nixtla) (they/them)

mlforecast_xgb_issue.ipynb

👀 1

Ginger Holt

09/26/2022, 11:55 PM

How long does this usually take to run? Generating the fitted values seems to take a long time, and I ended up canceling.

Copy code

fcst = StatsForecast(df=Y_df_train, 
                     models=[ETS(season_length=4, model='ZZA')], 
                     freq='QS', n_jobs=-1)
Y_hat_df = fcst.forecast(h=8, fitted=True)
Y_fitted_df = fcst.forecast_fitted_values()

https://github.com/Nixtla/hierarchicalforecast/blob/main/nbs/examples/AustralianDomesticTourism.ipynb

Ginger Holt

09/28/2022, 10:29 PM

I am running: reconcilers = [ BottomUp(), MinTrace(method='mint_shrink'), MinTrace(method='ols') ] hrec = HierarchicalReconciliation(reconcilers=reconcilers) Y_rec_df = hrec.reconcile(Y_hat_df, Y_fitted_merged_df, S, tags) and getting this error: KeyError: 'None of [RangeIndex(start=0, stop=11553, step=1)] are in the [index]' Any pointers?

Kin Gtz. Olivares

09/28/2022, 10:42 PM

Hi @Ginger Holt The hrec.reconcile method needs the Y_hat_df to be indexed by ‘unique_id’. For the moment the pd.DataFrame management of the library is problematic I am opening and Github issue to either add the indexing requirements to the documentation or perform the indexing within the methods.

Ginger Holt

09/29/2022, 5:17 PM

I am running: Y_hat_df = Y_hat_df.reset_index().set_index('unique_id') Y_fitted_merged_df = Y_fitted_merged_df.reset_index().set_index('unique_id') reconcilers = [ BottomUp(), MinTrace(method='mint_shrink'), MinTrace(method='ols') ] hrec = HierarchicalReconciliation(reconcilers=reconcilers) Y_rec_df = hrec.reconcile(Y_hat_df, Y_fitted_merged_df, S, tags) and getting this error: LinAlgError: Array must not contain infs or NaNs There are no infs or NaNs in the data. I am getting "False" for all of these checks: np.isinf(Y_fitted_merged_df).any() np.isinf(Y_hat_df).any() np.isnan(Y_fitted_merged_df).any() np.isnan(Y_hat_df).any() Data size is ~40K rows. So it may be a matrix computation issue at that scale. Have others gotten this working at the same scale?

Max (Nixtla)

09/29/2022, 5:19 PM

@Ginger Holt, I think, for some reason, the error was not included in your message.

Valeriy

10/06/2022, 3:43 PM

Hi Nixtla team, I am planning to deep dive into library what top 3-5 most performant models you would recommend me to look into @Max (Nixtla) @fede @Kin Gtz. Olivares

J T

10/06/2022, 9:14 PM

Hi Team Nixtla... got a question.. i was able to run the statsforecast. but the results came back funny - one month very high and the next month very low.. but the historical actual is not like that.. and my run on using pmdarima is very good. One thing i noticed is that the output DS was pointing to month end, instead of 1st day of the month.. not sure if that matters.

J T

10/06/2022, 9:15 PM

Copy code

Ran under AZ databrick.
here is the code: 
#Select SARIMA with seasonality 12
autoARIMA = AutoARIMA(season_length=12)

# Select ETS with seasonality 12 and multiplicative trend
model = StatsForecast(df=products2.set_index('unique_id'), 
                      models=[autoARIMA],
                      freq='m', n_jobs=-1)
Y_hat_df = model.forecast(horizon).reset_index()

J T

10/06/2022, 9:16 PM

here is the output results: unique_id ds AutoARIMA 0 0 2022-01-31 410.899323 1 0 2022-02-28 62994.761719 2 0 2022-03-31 493.694214 3 0 2022-04-30 65027.785156 4 0 2022-05-31 484.681763

J T

10/06/2022, 9:20 PM

original data is monthly volumes data (counts) ,with numeric unique_id ds y 2021-10-01 49941 2021-11-01 49731 2021-12-01 61980

Valeriy

10/09/2022, 11:34 AM

https://twitter.com/ChristophMolnar/status/1579051843107184641?s=20&t=moJyU55p_TECAzmelJ_SkQ this is way to funny

🤣 1

Max (Nixtla)

10/10/2022, 2:56 PM

Complex Exponential Smoothing (CES) is now avaialable in StatsForecast! https://www.linkedin.com/posts/mergenthaler_timeseries-forecasting-python-activity-69[…]029659398144-rlS8?utm_source=share&utm_medium=member_desktop

👏 3

Valeriy

10/11/2022, 9:28 AM

@Max (Nixtla) @Kin Gtz. Olivares @fede as mentioned to Max before I think this could be interesting for Nixtla to consider https://github.com/AbdullahO/mSSA. These models have been around for long time and MIT had a NeurIPS paper couple of years back. Also known as Caterpillar method due to the way how time series is turned into matrix and then decomposed there are several papers on it also books (using R) and also some R packages https://www.gistatgroup.com/cat/index.html I had tried it before on some limited electricity prices datasets and got good results, have not tried on retail data yet.

👀 1

Valeriy

10/11/2022, 9:29 AM

The MIT group that produces the library and NeurIPS paper does a lot of interesting work in forecasting.

Valeriy

10/11/2022, 9:30 AM

https://abdullaho.me/publication/mssa/

J T

10/14/2022, 6:10 PM

trouble installing numba, statsmodels and statsforecst. it errors out today be never last week. something has changed? %pip install -U numba Python interpreter will be restarted. WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f5fdd406fd0>: Failed to establish a new connection: [Errno 101] Network is unreachable')': /simple/numba/

👀 1

Max (Nixtla)

10/14/2022, 6:55 PM

Have you tried using the defined environment:

Copy code

<https://github.com/Nixtla/statsforecast/blob/main/environment.yml>

Max (Nixtla)

10/14/2022, 6:58 PM

Something like:

Copy code

conda env create --name statsforecastenv --file=environment.yml

And then

Copy code

source activate statsforecastenv

Valeriy

10/17/2022, 12:58 PM

Hi team tried to run Colab with transfer learning from Nixtla github over weekend but got error. https://colab.research.google.com/drive/1uFCO2UBpH-5l2fk3KmxfU0oupsOC6v2n?authuser=0&pli=1#scrollTo=I4eQDyW08bjm how can one fix this please?

J T

10/23/2022, 2:31 AM

Hi team, ran StatsForecast AutoArima and got an error, wonder how we can see which unique id triggers the error to examine what was going? here is the error log: RemoteTraceback Traceback (most recent call last) RemoteTraceback: “”" Traceback (most recent call last): …… ValueError: order must be 3 non-negative integers, got (0, 0, 0) “”" The above exception was the direct cause of the following exception: