PySpark Usage Guide for Pandas with Apache Arrow
The Arrow usage guide is now archived on this page.
For opt-in execution inside the executor JVM, see the in-process Python UDF guide, including its native dependency requirements and shared-process limitations.