Vectorized UDF: Scalable Analysis with Python and PySpark - Li Jin

One of the best videos on this topic that i have seen. Thanks!

arent pandas UDF working on multiple rows? please correct me if im wrong.

Thanks for great talk! When using grouped map pandas udf for model training I am assuming that your group_column=id is unique. This doesn't hurt the performance since it needs to groupby unique items ?

An essential great talk.

Great video. Thanks so much. I am using pandas udf as a solution, where I ran into severe memory issues because of the serialization involved using python objects. I am using grouped map pandas udf and it expects static return type and that much poses a challenge in writing generic functions that can be decorated with pandas udf. Is there a way i can infer the return type during run time while using pandas udf?