I am trying to bucketize columns that contain the word "road" in a 5k dataset. And create a new dataframe.
I am not sure how to do that, here is what I have tried far :
from pyspark.ml.feature import Bucketizer
spike_cols = [col for col in df.columns if "road" in col]
for x in spike_cols :
bucketizer = Bucketizer(splits=[-float("inf"), 10, 100, float("inf")],
inputCol=x, outputCol=x + "bucket")
bucketedData = bucketizer.transform(df)