User Guide

Welcome to the PySpark user guide! Each of the below sections contains code-driven examples to help you get familiar with PySpark.

• Chapter 1: DataFrames - A view into your structured data
  • Create a DataFrame
  • View the DataFrame
  • DataFrame Manipulation
  • DataFrames vs. Tables
  • Save DataFrame to Persistent Storage
  • Native DataFrame Plotting
• Chapter 2: A Tour of PySpark Data Types
  • Basic Data Types in PySpark
  • Precision for Doubles, Floats, and Decimals
  • Complex Data Types in PySpark
  • Casting Columns in PySpark
  • Semi-Structured Data Processing in PySpark
• Chapter 3: Function Junction - Data manipulation with PySpark
  • Clean data
  • Transform data
  • Summarizing data
  • When DataFrames Collide: The Art of Joining
• Chapter 4: Bug Busting - Debugging PySpark
  • Spark UI
  • Monitor with top and ps
  • Use PySpark Profilers
  • Display Stacktraces
  • Python Worker Logging
  • IDE Debugging
• Chapter 5: Unleashing UDFs & UDTFs
  • Python UDFs
  • Python UDTFs
• Chapter 6: Old SQL, New Tricks - Running SQL on PySpark
  • Introduction
  • Running SQL with PySpark
  • SQL vs. DataFrame API in PySpark
  • Using SQL and DataFrame API Interchangeably
• Chapter 7: Load and Behold - Data loading, storage, file formats
  • Reading Data
  • Writing Data
  • Additional Options and Configurations
• ANSI Migration Guide - Pandas API on Spark
  • Behavior Change
  • Related Configurations