How can the scikit-learn quantile transformer module be used to pre-process river stream data?

[medha-chippa]

The Scikit-learn Quantile Transformer module is being used to pre-process river stream object. The following is the piece of code that attempts to do this:

c1 = ensemble.AdaptiveRandomForestClassifier(split_confidence=0.01,tie_threshold=0.07, seed=123)

c2= multiclass.OneVsOneClassifier(ensemble.AdaptiveRandomForestClassifier(split_confidence=0.01 ,tie_threshold=0.06, seed=123))

from sklearn.preprocessing import QuantileTransformer

scaler1 = QuantileTransformer(n_quantiles=3, random_state=123)
scaler2 = QuantileTransformer(n_quantiles=3, output_distribution='normal', random_state=456)

scaler1.fit(x)
scaler2.fit(x)

strm_st1=iter_pandas(X=x , y=y)
strm_st2=iter_pandas(X=x , y=y)

metric1 = metrics.Accuracy()
metric2 = metrics.Accuracy()

y_preds_st1 = []
y_preds_st2 = []

for ((xi1, yi1),(xi2, yi2)) in zip(strm_st1,strm_st2):

x1_trans=scaler1.transform(pd.DataFrame([xi1])) #converting dictionary to pandas dataframe and transforming using scaler object. x1_trans is of type int

x2_trans=scaler2.transform(pd.DataFrame([xi2]))

ar_yi1=np.array(yi1.item()) #converting int64 type data to int and converting the result to an array

ar_yi2=np.array(yi2.item())

ar_yi1.shape=(1,1)

ar_yi2.shape=(1,1)

dt1_i=iter_array(x1_trans,ar_yi1)

dt2_i=iter_array(x2_trans,ar_yi2)

for ((xi1_c,ar_yi1),(xi2_c,ar_yi2)) in zip(dt1_i,dt2_i):

`y_pred1=c1.predict_one(xi1_c)`

`y_pred2=c2.predict_one(xi2_c)`

`c1.learn_one(xi1_c,` ar_yi1) #error generated here

`c2.learn_one(xi2_c,` ar_yi2)

`metric1.update(y_pred=y_pred1, y_true=ar_yi1)`

`metric2.update(y_pred=y_pred2, y_true=ar_yi2)`

`y_preds_st1.append(y_pred1)`

`y_preds_st2.append(y_pred2)`

print(f"Accuracy for c1: {metric1.get()}")

print(f"Accuracy for c2: {metric2.get()}")

The above piece of code results in an error: TypeError: unhashable type: 'dict'

How can this be rectified?

Thank you

[MaxHalford]

I'm not sure what to say. You're passing a pandas.DataFrame to a River scaler. River objects expect to be passed dictionaries, not DataFrames. Isn't this clear from the documentation? I'm really at a loss here... You don't seem to be putting a lot of effort into understanding the library.

Meta: please format the code property and make it reproducible. Please use GitHub's Markdown syntax to highlight the code. Every post you make is of very low quality and makes it frustrating to answer.

Meta 2: please do not open a discussion every time you have a question. It's polluting the Discussions tab. Just keep the questions in a single discussion.

Frankly, I think you should spend more time reading the documentation and the examples therein. You seem to have a fundamental lack of understand of 1) Python 2) The approach proposed by River. Don't take me wrong: it's fine to be a beginner and not to know these things. But I seriously advise you to spend some more time understanding what you're doing instead of asking us open questions such as these. You won't progress like this.

[medha-chippa]

The StandardScaler is an sklearn.preprocessing module and not a river scaler, a pandas dataframe has hence been given to it as input (and not a dictionary). The dictionary being referenced in the stream object is being converted to a dataframe and then given to scikit learn's standard scaler as input. Relevant comments explaining the same have been included

Please read through the code carefully. The relevant libraries have been imported to reflect the use of a scikit learn scaler.

Also, I shall make sure that related questions are nested within one discussion from now on and the code snippets are legible. I apologize for the inconvenience caused.

Thank you

[medha-chippa]

The idea here is to be able to use the Quantile Transformer from sklearn.preprocessing. The code has been modified to reflect the same (Standard Scaler replaced with Quantile Transformer).

Please help me rectify this error.

Thank you

[MaxHalford]

Look, it's plain and simple: you can't use scikit-learn with River. It's just not meant to be!

If you want, you can preprocess your data with a quantile transformer from scikit-learn before running the data through a River pipeline, but that defeats the whole point of online learning.

Please help me rectify this error.

I really don't have enough context to help you. Your code is not runnable as is. The error message you give doesn't even indicate at what line the error is occurring. You're really not make any of our lives easier here 😅

[medha-chippa]

Please let me know if including an image of the error trace and the code could be of any help. Also, the import statement for importing a scikit learn scaler is on line no 3(it has been there since the beginning of the discussion). The idea is to be able to preprocess data in an incremental fashion, as and when the data arrives from the river stream.

Thank you

[MaxHalford]

You can't use scikit-learn's QuantileTransformer to process data online. Period.

With regards to your code, I still can't run it. As a maintainer of an opensource library, I have little time to spare. I don't think it's too much to ask for a tidy and formatted code block which I can copy paste in my interpreter. I can't do that with your code. For instance, what is x?

Please, look at other issues and discussions and River: they're much more helpful. You're really not doing yourself any favors.

[medha-chippa]

I apologize. Sorry for the inconvenience. I shall try to figure this out myself.

Thank you.