nixtlacommunity #hierarchicalforecast

Channels

random

linen

statsforecast

TWINCY ZATAKIA

06/18/2023, 4:16 PM

Hello! I'm trying to use hierarchical forecast for multivariate time series model, but I'm not able to find multivariate time series examples anywhere. Can anyone help to find out?

Harshita Ved

06/19/2023, 7:09 PM

Hi! I'm trying to use hierarchical forecast for my dataset and unable to generate tags. Do we have an example with a regular dataframe as dataset and tags generated explicitly before hierarchical forecast?

Kin Gtz. Olivares

06/20/2023, 1:02 AM

Hi @Jeff Tackes, It is my impression that this error arises from HierarchicalForecast assuming that the inputs are "balanced", ie you have the same number of observations for all the base predictions. The reshape fails because the

Y_hat_df

's observations are not a multiple of your base predictions.

Jeff Tackes

06/20/2023, 2:28 AM

Thank you Kin! Yes that was it 🙂 i was missing some forecast dates for a few id's and it caused the unbalance as you were saying

Isaac

06/21/2023, 2:24 PM

What's the best way to integrate MLForecast and Hierarchicalforecast? If I want to use a lot of the Hierarchicalforecast methods, I need the fitted values, but MLForecast doesn't have a method to get them.

Harshita Ved

06/22/2023, 3:41 PM

Hi, During reconciliation I am facing a tags mismatch error, I think it is from an exception in core.py .

reconciled_forecasts = hrec.reconcile(Y_hat_df=Y_hat_df, Y_df=hts_orders_agg, S=hts_S_orders, tags=hts_orders_tags)

Error:

Copy code

Exception: Check `S_df`, `Y_hat_df` series difference, S\Y_hat=50, Y_hat\S=0

I also tried passing the same df in

Y_hat_df

and

Y_df

just to ensure the tags are exactly the same. Apparently it always gives a mismatch error. I created tags on overall dataset and afterwards splitted in train-test just like examples in the package. Unable to move forward. Has anyone faced the same issue and know solution?

Jack Solomon

07/29/2023, 12:32 AM

Hey all! I’m trying to get hierarchicalforecast working with already-done XGBoost forecasts. In other words, I’ve got forecasts for each unique ID at each level on each day in my test set - is it possible to now just throw them into hierarchicalforecast with the summing matrix?

Nasreddine D

07/31/2023, 6:18 PM

Hi, I have a dataset with the following columns (It's a B2B business) : • Clients_Id • Stores_Id • Product_Id • Date (working days) • Sales (quantity sold) The daily data is intermittent and I am asked to forecast the quantity of each single products for each client, from each stores. We don't need to forecast at higher level (ex: total of all sales). But I was wondering if I treat the problem as hierarchical forecasting, it could help to have a better accuracy at the lowest level? Or just a global model at the lowest level should be just fine?

Ada Puślecka

08/25/2023, 11:10 AM

Hi, is it possible to make hierarchical forecasting if constituent series have different lengths - for example they finish earlier or they start later than the rest? My goal is to forecast the sum of around 100 series, however the amount of these series might change - new ones could join or some of the existing ones might finish which will cause the aggregated forecast have changes in the level of values. I am wondering if there exist methods for this kind of scenarios or do I just need to fill the rest of the values for all the series even if this would be a very long periods to be filled?

➕ 2

👀 1

Chad Parmet

09/15/2023, 1:25 PM

Thank you for this package! Following up on @Ada Puślecka’s question, does

HierarchicalForecast

allow for variable length time series, or does it assume that all of the time series/levels have complete data for the the full maximum time frame in the dataset? For example say I have a hierarchical dataset with sales by store, nested into countries, over the past 12 months at max. Does HF assume every time series will have 12 complete months of data? Or does it handle appropriately a dataset where one time series (store) has only 3 months of data, one of the countries may have only 6 months etc? I'm not getting errors when I run this kind of dataset, but I'm not sure if it explains why results are unexpected Thank you!

➕ 1

❤️ 1

Jemima Graham

09/19/2023, 9:27 AM

Hi, can the hierarchicalforecast package be used for temporal hierarchies (like thief in R)? Thanks!

👍 1

Jason Gofford

10/13/2023, 4:01 PM

I don't have a repro for it yet, but any idea why I would be getting this

TypeError: unhashable type: 'set'

error when calling

reconcile

? Full stack trace in thread.

Jason Gofford

10/14/2023, 8:27 PM

Is there a way to avoid

aggregate

from returning 0 valued series for missing hierarchy combinations? As an example, if I have two countries, let's say France and Germany, I will have items sold in France but not German. From a conceptual perspective it doesn't make sense to have series corresponding to French items in the German market (even if they're 0) as they don't exist. As it stands the

aggregate

function is causing a 20x explosion in rowcount in my dataset, mostly due to 0 sum series of missing combinations

Francisco

10/17/2023, 3:30 AM

hey I have one question. I understand that hierarchicalforecast serves as a way to reconcile time series that sum up to each other (like the sum of the GDP of a country is the sumof the GDP of its states, cities, etc) but does it also work to reconcile time series of lets say time granularity? like the hourly time of a stock, the daily price, weekly and monthly? does that make sense? why would anyone want to do that ?

Jason Gofford

10/18/2023, 1:22 PM

For hierarchical reconciliation, what is the recommended way to handle series that only appear in

Y_test_df

because their length is lower than my horizon? At the moment, if I use

aggregate

to create the hierarchy I end up in a "series differences" scenario because my

S_df

has levels that do not exist in

Y_train_df

(as they're only in

Y_test_df

). I can filter

S_df

to only include what's in

Y_train_df

but then I think I lose the ability to extract the test-set forecasts for those shorter series. Is there a better way?

➕ 1

Francisco

11/07/2023, 1:34 PM

Hello, I have a question

Francisco

11/07/2023, 1:34 PM

Copy code

C:\Users\M330618\.conda\envs\Merck\lib\site-packages\hierarchicalforecast\methods.py:261: RuntimeWarning:

invalid value encountered in divide

C:\Users\M330618\.conda\envs\Merck\lib\site-packages\hierarchicalforecast\methods.py:687: UserWarning:

Replacing negative forecasts with zero.

---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
Cell In[544], line 19
     16 rec_model = HierarchicalReconciliation(reconcilers=reconcilers)
     18 #crear matriz de reconciliación
---> 19 p_rec = rec_model.reconcile(Y_hat_df=forecast_agg, Y_df=train_agg, S=S_train, tags=tags)

File ~\.conda\envs\Merck\lib\site-packages\hierarchicalforecast\core.py:265, in HierarchicalReconciliation.reconcile(self, Y_hat_df, S, tags, Y_df, level, intervals_method, num_samples, seed, sort_df)
    261     fcsts_model = reconciler.predict(S=reconciler_args['S'], 
    262                                      y_hat=reconciler_args['y_hat'], level=level)
    263 else:
    264     # Memory efficient reconciler's fit_predict
--> 265     fcsts_model = reconcile_fn(**kwargs, level=level)
    267 # Parse final outputs
    268 Y_tilde_df[recmodel_name] = fcsts_model['mean'].flatten()

File ~\.conda\envs\Merck\lib\site-packages\hierarchicalforecast\methods.py:761, in MinTrace.fit_predict(self, S, y_hat, idx_bottom, y_insample, y_hat_insample, sigmah, level, intervals_method, num_samples, seed, tags)
    758         raise Exception('idx_bottom needed for nonnegative reconciliation')
    760 # Fit creates P, W and sampler attributes
--> 761 self.fit(S=S,
    762          y_hat=y_hat,
    763          y_insample=y_insample,
    764          y_hat_insample=y_hat_insample,
    765          sigmah=sigmah,
    766          intervals_method=intervals_method,
    767          num_samples=num_samples,
    768          seed=seed,
    769          tags=tags, idx_bottom=idx_bottom)
    771 return self._reconcile(S=S, P=self.P, y_hat=self.y_hat,
    772                        level=level, sampler=self.sampler)

File ~\.conda\envs\Merck\lib\site-packages\hierarchicalforecast\methods.py:706, in MinTrace.fit(self, S, y_hat, y_insample, y_hat_insample, sigmah, intervals_method, num_samples, seed, tags, idx_bottom)
    703 bottom_fcts = np.apply_along_axis(lambda y_hat: solve_qp(G=G, a=a @ y_hat, C=C, b=b)[0], 
    704                                   axis=0, arr=y_hat)
    705 if not np.all(bottom_fcts > -1e-8):
--> 706     raise Exception('nonnegative optimization failed')
    707 # remove negative values close to zero
    708 bottom_fcts = np.clip(np.float32(bottom_fcts), a_min=0, a_max=None)

Exception: nonnegative optimization failed

Francisco

11/07/2023, 1:35 PM

when trying to run my reconciliation I get the following error: Exception: nonnegative optimization failed. I dont get it, i checked and there are no negative values either in my train or forecasts df

Francisco

11/07/2023, 1:37 PM

I have to say I did mlforecast and statsforecast crossvalidationm then joined the two and got the best models and best forecasts to pass through the reconciliation, maybe this can be the problem?

Yacoub Alatrash

11/09/2023, 8:14 PM

Hey guys. We are exploring the opportunity of using your hierarchical forecasting package in our prod at Wayfair. I have a question: is there a way to parallelize the code for the OLS reconciliation part?

Yacoub Alatrash

11/10/2023, 7:36 PM

For example: the hts package on R has a "parallel" option, if TRUE it imports parallel package to allow parallel processing but not sure if something similar is encoded into your python package.

Toni Borders

11/14/2023, 4:54 PM

Is the hierarchicalforecast compatible with macs (M2 chip)? I am getting the following error when trying to run the example. It appears to be error immediately at the import of the methods.

Traceback (most recent call last):

File “/Users/tmb/PycharmProjects/data-science/UFE/code/hierarchFrcstEx.py”, line 14, in <module>

from hierarchicalforecast.methods import BottomUp, TopDown, MiddleOut

File “/Users/tmb/Library/Caches/pypoetry/virtualenvs/data-science-zw503lJq-py3.10/lib/python3.10/site-packages/hierarchicalforecast/methods.py”, line 14, in <module>

_from quadprog import solve_qp_

_ImportError: dlopen(/Users/tmb/Library/Caches/pypoetry/virtualenvs/data-science-zw503lJq-py3.10/lib/python3.10/site-packages/quadprog.cpython-310-darwin.so, 0x0002): tried: ‘/Users/tmb/Library/Caches/pypoetry/virtualenvs/data-science-zw503lJq-py3.10/lib/python3.10/site-packages/quadprog.cpython-310-darwin.so’ (mach-o file, but is an incompatible architecture (have ‘x86_64’, need ‘arm64’)), ‘/System/Volumes/Preboot/Cryptexes/OS/Users/tmb/Library/Caches/pypoetry/virtualenvs/data-science-zw503lJq-py3.10/lib/python3.10/site-packages/quadprog.cpython-310-darwin.so’ (no such file), ‘/Users/tmb/Library/Caches/pypoetry/virtualenvs/data-science-zw503lJq-py3.10/lib/python3.10/site-packages/quadprog.cpython-310-darwin.so’ (mach-o file, but is an incompatible architecture (have ‘x86_64’, need ‘arm64’))_

Toni Borders

11/21/2023, 12:07 PM

I am trying to run through this example in the documentation with my data, but am hitting the follow error.

Traceback (most recent call last):

File “/Users/tmb/PycharmProjects/data-science/UFE/code/hierFcst.py”, line 106, in <module>

_main(data, freq, metadata_str, account)_

File “/Users/tmb/PycharmProjects/data-science/UFE/code/hierFcst.py”, line 88, in main

_Y_df = aggregate(Y_df, spec)_

File “/Users/tmb/PycharmProjects/data-science/UFE/code/hierFcst.py”, line 81, in aggregate

_Y_df, S_df, tags = aggregate(Y_df, spec)_

File “/Users/tmb/PycharmProjects/data-science/UFE/code/hierFcst.py”, line 81, in aggregate

_Y_df, S_df, tags = aggregate(Y_df, spec)_

File “/Users/tmb/PycharmProjects/data-science/UFE/code/hierFcst.py”, line 81, in aggregate

_Y_df, S_df, tags = aggregate(Y_df, spec)_

[Previous line repeated 995 more times]

RecursionError: maximum recursion depth exceeded

Here is my code:

Copy code

def get_spec():
    spec = [
        ['org'],
        ['org', 'account'],
        ['org','meter'],
        ['org','account','meter']
    ]
    return spec

def aggregate(Y_df, spec):
    Y_df, S_df, tags = aggregate(Y_df, spec)
    Y_df = Y_df.reset_index()

def main(data):
    Y_df = clean_data(data)
    spec = get_spec()
    Y_df = aggregate(Y_df, spec)
    print(Y_df.head(20))

Any thoughts on what could be going wrong?

Graeme Davidson

11/22/2023, 4:28 PM

Hi there, I'm working on a project looking at hierarchal forecasting, I'm looking at darts and obviously this package. One thing I have noticed is that none of the examples use exogenous variables, like price, promotions, events etc. I assume the models can handle these, but before I went blundering in I wondered if anyone had any good examples of setting up say an hierarchal model that uses exogenous variables. Do we need to do anything specific with different exogenous variables? I appreciate any thoughts or suggestions you might have, thank you.

11/25/2023, 4:24 PM

Hello, I have the following scenario and was wondering if there is an easier way to do what I want to do than the one I am thinking of. Imagine I have a dataset with the following columns. ds(time) | A(cat) | B(cat) | C(cat) | Total(num) | X(num) | X_percentage(num) = X/Total Eventually, I want to generate ‘X_percentage’ forecasts for each unique categorical combination, so for each unique combination 'A-B-C' I want to provide X_percentage for future times. Until here I could just use any model of any of your forecast methods (stats, ml , neural), and that is it. My concern is if I want to apply hierarchical reconciliation to those forecasts. If I had actual volumes (like X(num)) I could just apply one of your hierarchical forecasting algorithms which aggregates (sum) the target. However, the fact that I have percentages I don’t think I can either sum those or even average them to upper granularities. Can you think of any way to apply your hierarchical reconciliation library for a case like this? The solution I can think of would be to separately forecast ‘X(num)’ and ‘Total’ by any forecast model and apply hierarchical reconciliation separately on them. After that, I would merge those two by ‘unique_id’, and finally calculate X_percentage(num) as X/Total. Wondering though if there is an easier way :). Thanks in advance.