Hey <@U0316L4HWQ7>, how do you manage input `DataF...
# generala
Akmal Soliev
03/11/2023, 1:57 PMHey @fede (nixtla) (they/them), how do you manage input
DataFrames👍 1
🙌 1
f
fede (nixtla) (they/them)
03/13/2023, 5:16 PMHey @Akmal Soliev! That would be an amazing enhancement. The function that transforms the dataframe into numpy is this: https://github.com/Nixtla/statsforecast/blob/main/statsforecast/core.py#L407.
a
Akmal Soliev
03/14/2023, 10:08 AMThx, will take a look!
Akmal Soliev
03/14/2023, 8:39 PMHey, I had a follow up question, I'm currently trying to use your synthetic data generator function (https://github.com/Nixtla/statsforecast/blob/main/statsforecast/utils.py). In order to use it as polars DataFrame I would need to recreate the steps from scratch due to the categorical dtypes (I have no clue why that's such a pain in the @$$ to deal with). Prior to doing that I wanted to ask you what's the point of having those values as categorical?
f
fede (nixtla) (they/them)
03/14/2023, 8:54 PMhey @Akmal Soliev! It is just for efficiency purposes when dealing with experiments that have thousands of time series, it’s more memory efficient to deal with categorical instead of object with such a large amount of data
fede (nixtla) (they/them)
03/14/2023, 8:56 PMmaybe we could add a new argument to
generate_seriesuid_to_categorical=Truea
Akmal Soliev
03/15/2023, 11:17 AMI'll see if I can just rebuild it
Akmal Soliev
03/15/2023, 4:53 PMokay rebuilt it, also got some time shaved of the generation:
Copy code
before:
________________________________________________________
Executed in 3.93 secs fish external
usr time 4.68 secs 0.06 millis 4.68 secs
sys time 1.44 secs 1.03 millis 1.43 secs
after:
________________________________________________________
Executed in 833.23 millis fish external
usr time 2.23 secs 0.07 millis 2.23 secs
sys time 0.50 secs 1.20 millis 0.50 secsAkmal Soliev
03/15/2023, 5:53 PMThis is the most useless improvement 🤣
m
Max (Nixtla)
03/17/2023, 6:41 PM@Akmal Soliev we think its great. Thanks.
🙏 1
a
Akmal Soliev
05/09/2023, 9:21 PM[PR Update] After two months of persistent effort, I have successfully developed a versatile DataFrame conversion solution for both Polars and Pandas. This solution is compatible with any structured two-dimensional data, provided that the output can be consolidated into named Numpy arrays.
✅ All local tests have been passed, Github Actions required
util.py_StatsForecastStatsForecastcore.ipynbgenerate_series🙌 3
f
fede (nixtla) (they/them)
05/10/2023, 7:58 PMAwesome @Akmal Soliev!
m
Max (Nixtla)
05/10/2023, 8:09 PMYou are the best @Akmal Soliev, thanks.
Max (Nixtla)
05/10/2023, 8:10 PMWe are going to write a brief post to communicate the new feature. Which accounts should we mention to thank you.
a
Akmal Soliev
05/11/2023, 3:58 PM@Max (Nixtla) Thank you, I'm glad to help.
I just implemented a change where if input dataframe is polars or pandas
StatsForecastplotself.enginem
Max (Nixtla)
05/11/2023, 7:26 PMGreat :)
a
Akmal Soliev
05/12/2023, 11:01 PM@Max (Nixtla)
All the local tests have been passed on Polars and Pandas using modified
generate_seriesAkmal Soliev
05/14/2023, 8:32 PM@fede (nixtla) (they/them) here's the file with all the tests done with Polars.
NOTE:
groupby.sort('unique_id')Akmal Soliev
05/23/2023, 3:20 PMHey, there is a
_parse_ds_typemainintissubclass(df["ds"].dtype.type, np.integer)np.array().dtype.kind in ["i", "f"]if🙌 1
2 Views