# PySpark Cheat Sheet ## Introduction PySpark is the Python API for Apache Spark, a unified analytics engine designed for large-scale data processing and distributed computing across clusters. It combines the ease and flexibility of Python with Spark's powerful distributed computing capabilities, enabling data engineers and scientists to process massive datasets efficiently through in-memory computation, lazy evaluation, and a rich ecosystem of APIs including DataFrames, SQL, streaming, and machine learning (MLlib). This comprehensive cheat sheet covers essential PySpark concepts from basic DataFrame operations and transformations to advanced topics like performance optimization, streaming, UDFs, and cluster configuration, providing practical examples, use cases, advantages, considerations, and authoritative references organized into logical categories for efficient learning and production implementation. --- ## Table 1: SparkSession and Configuration | Technique/Concept | Description | Main Usages | Advantages | Considerations | Example | Reference | |---|---|---|---|---|---|---| | SparkSession | Entry point to PySpark functionality, creates DataFrame and executes SQL | Application initialization, resource management, configuration | Unified API for all Spark functionality, builder pattern, lazy initialization | Must be created first, only one active per application recommended | `from pyspark.sql import SparkSession`
`spark = SparkSession.builder`
` .appName("MyApp")`
` .getOrCreate()` | 1, 2 | | spark.conf.set() | Set configuration properties at runtime | Dynamic configuration, tuning, feature toggles | Runtime configuration, no restart needed | Some configs cannot be changed after initialization | `spark.conf.set("spark.sql.shuffle.partitions", "200")`
`spark.conf.set("spark.executor.memory", "4g")` | 3, 4 | | Executor Memory | Memory allocated per executor for task execution | Cluster configuration, memory tuning | Controls memory available for computation, caching | Must balance with number of executors, too much can waste resources | `spark = SparkSession.builder`
` .config("spark.executor.memory", "8g")`
` .getOrCreate()` | 4, 5, 6 | | Executor Cores | Number of CPU cores per executor | Parallelism control, resource allocation | Controls task parallelism per executor | More cores = fewer executors, balance needed | `spark = SparkSession.builder`
` .config("spark.executor.cores", "4")`
` .getOrCreate()` | 4, 5, 6 | | Driver Memory | Memory for driver program | Large collects, broadcast variables, metadata | Driver-side operations, storing results | Cannot be set at runtime, must configure at startup | `spark = SparkSession.builder`
` .config("spark.driver.memory", "4g")`
` .getOrCreate()` | 4, 5 | | Deployment Modes | Client vs Cluster mode for driver location | Production deployment, resource management | Client=driver on submit machine, Cluster=driver on cluster | Client mode easier for development, Cluster for production | `# Client mode (default)`
`spark-submit --deploy-mode client app.py`
`# Cluster mode`
`spark-submit --deploy-mode cluster app.py` | 7, 8 | | Local Mode | Run Spark on single machine with threads | Development, testing, debugging | No cluster needed, easy setup, fast iteration | Not for production, limited by single machine resources | `spark = SparkSession.builder`
` .master("local[*]")`
` .getOrCreate()` | 7, 8 | | Shuffle Partitions | Number of partitions for shuffle operations | Performance tuning, groupBy/join optimization | Controls parallelism after shuffles, default 200 | Too few=large partitions, too many=overhead | `spark.conf.set("spark.sql.shuffle.partitions", "400")` | 3, 9 | --- ## Table 2: DataFrame Creation and Data Sources | Technique/Concept | Description | Main Usages | Advantages | Considerations | Example | Reference | |---|---|---|---|---|---|---| | createDataFrame() | Create DataFrame from Python collections or RDDs | Testing, small datasets, manual creation | Simple, no external files needed, good for examples | Not for large data, not distributed source | `data = [("Alice", 25), ("Bob", 30)]`
`df = spark.createDataFrame(data, ["name", "age"])` | 10, 11 | | read.csv() | Read CSV files into DataFrame | Data ingestion, ETL, file processing | Common format, header/schema inference, delimiter options | Slower than Parquet, schema inference expensive | `df = spark.read.csv("data.csv", header=True, inferSchema=True)`
`df = spark.read.option("delimiter", ";").csv("data.csv")` | 12, 13, 14 | | read.parquet() | Read Parquet columnar format files | Production data loading, optimized storage | Columnar, compressed, schema embedded, very fast | Binary format, not human-readable | `df = spark.read.parquet("data.parquet")`
`df = spark.read.parquet("data/*.parquet")` | 12, 13, 15 | | read.json() | Read JSON files into DataFrame | API data, nested structures, semi-structured data | Handles nested data, schema inference, flexible | Slower than Parquet, nested schemas can be complex | `df = spark.read.json("data.json")`
`df = spark.read.option("multiline", True).json("nested.json")` | 12, 13, 16 | | read.table() / read.text() | Read text files line by line | Log files, unstructured text, custom parsing | Simple text reading, good for custom formats | Returns single column, manual parsing needed | `df = spark.read.text("logs.txt")`
`df = spark.read.table("database.table_name")` | 12, 13 | | read.jdbc() | Read from relational databases via JDBC | Database integration, ETL from RDBMS | Direct database connection, partitioned reads | Requires JDBC driver, network latency | `df = spark.read.format("jdbc")`
` .option("url", "jdbc:postgresql://host:5432/db")`
` .option("dbtable", "schema.table")`
` .option("user", "user").load()` | 12, 17 | | read.format() | Generic reader with format specification | Custom formats, Avro, ORC, Delta, etc. | Flexible, supports many formats | Requires format-specific options | `df = spark.read.format("avro").load("data.avro")`
`df = spark.read.format("delta").load("delta_table")` | 12, 13 | | Schema Definition (StructType) | Explicitly define DataFrame schema | Schema enforcement, performance, data validation | No schema inference overhead, type safety, validation | More verbose, must know schema beforehand | `from pyspark.sql.types import *`
`schema = StructType([`
` StructField("name", StringType(), True),`
` StructField("age", IntegerType(), False)`
`])`
`df = spark.read.schema(schema).csv("data.csv")` | 18, 19, 20 | | write.csv() / write.parquet() / write.json() | Write DataFrame to various file formats | Data export, ETL outputs, data lake storage | Multiple formats, partitioning, compression | Overwrites by default, choose mode carefully | `df.write.mode("overwrite").csv("output.csv")`
`df.write.partitionBy("year").parquet("output")`
`df.write.mode("append").json("output.json")` | 12, 13, 14 | | Save Modes | Overwrite, append, ignore, errorIfExists | Output control, idempotency, data safety | Flexible write behavior, prevents accidents | Overwrite deletes existing data | `df.write.mode("overwrite").save("path")`
`df.write.mode("append").save("path")`
`df.write.mode("ignore").save("path")` | 12, 13 | --- ## Table 3: DataFrame Transformations (Lazy) | Technique/Concept | Description | Main Usages | Advantages | Considerations | Example | Reference | |---|---|---|---|---|---|---| | select() | Select specific columns | Column projection, reducing data width | Simple column selection, renames with alias | Returns new DataFrame, original unchanged | `df.select("name", "age")`
`df.select(col("name").alias("full_name"))` | 21, 22 | | selectExpr() | Select with SQL expressions | Complex column calculations, SQL syntax | SQL-like syntax, expressions in strings | Strings less type-safe than col() objects | `df.selectExpr("name", "age * 2 as double_age")`
`df.selectExpr("upper(name) as upper_name")` | 23, 24 | | filter() / where() | Filter rows based on conditions | Data filtering, subsetting, cleaning | Readable conditions, chainable, synonyms | Broadcast join auto-detection can affect filter pushdown | `df.filter(col("age") > 25)`
`df.where("age > 25 AND name IS NOT NULL")` | 21, 22 | | withColumn() | Add or replace column | Feature engineering, column creation, transformation | Flexible, expressions, multiple calls chain | Creates new DataFrame, can be expensive if many withColumn calls | `df.withColumn("age_plus_10", col("age") + 10)`
`df.withColumn("category", when(col("age") > 30, "senior").otherwise("junior"))` | 21, 22, 25 | | withColumnRenamed() | Rename column | Schema adjustment, clarity, compatibility | Simple renaming, chainable | Only renames, doesn't transform | `df.withColumnRenamed("old_name", "new_name")` | 21, 22 | | drop() | Remove columns | Schema reduction, removing unnecessary fields | Clean schema, reduces memory | Cannot recover dropped columns without recomputation | `df.drop("column_name")`
`df.drop("col1", "col2", "col3")` | 21, 22 | | distinct() / dropDuplicates() | Remove duplicate rows | Deduplication, unique values | Simple deduplication, subset parameter | Can be expensive on large data, requires shuffle | `df.distinct()`
`df.dropDuplicates(["name", "age"])` | 21, 22 | | orderBy() / sort() | Sort DataFrame by columns | Ordering, ranking, display | Multiple columns, ascending/descending | Expensive operation, requires shuffle, avoid for large data unless necessary | `df.orderBy("age", ascending=False)`
`df.sort(col("age").desc(), col("name").asc())` | 21, 22 | | limit() | Limit number of rows returned | Sampling, testing, quick preview | Fast for small samples | Not random sampling, takes first n rows | `df.limit(10)` | 21, 22 | | alias() | Rename DataFrame (for joins) | Self-joins, disambiguation, readability | Clearer join syntax | Only for reference, not a transformation | `df1 = df.alias("df1")`
`df2 = df.alias("df2")`
`df1.join(df2, df1.id == df2.id)` | 21, 22 | | union() / unionAll() / unionByName() | Combine DataFrames vertically | Appending data, merging datasets | Stacks rows, union vs unionAll (no dedup vs dedup) | Columns must match (or use unionByName) | `df1.union(df2)`
`df1.unionByName(df2, allowMissingColumns=True)` | 21, 22, 26 | | join() | Join two DataFrames | Data merging, enrichment, combining sources | Multiple join types (inner, outer, left, right, semi, anti) | Can be expensive, optimize with broadcast | `df1.join(df2, df1.id == df2.id, "inner")`
`df1.join(df2, ["id"], "left_outer")` | 27, 28, 29 | | Broadcast Join | Optimize small table joins by broadcasting | Joining small dimension tables to large fact tables | No shuffle for small table, very fast | Only for small tables (<10MB default), manual or automatic | `from pyspark.sql.functions import broadcast`
`large_df.join(broadcast(small_df), "key")` | 28, 29, 30 | --- ## Table 4: DataFrame Actions (Eager) | Technique/Concept | Description | Main Usages | Advantages | Considerations | Example | Reference | |---|---|---|---|---|---|---| | show() | Display DataFrame rows | Debugging, data exploration, verification | Shows formatted table, limit rows | Collects data to driver, avoid on large data | `df.show()`
`df.show(20, truncate=False)` | 21, 22 | | count() | Count number of rows | Data validation, metrics, logging | Simple count, returns long | Triggers full scan, can be slow | `count = df.count()`
`print(f"Rows: {count}")` | 21, 22 | | collect() | Retrieve all rows to driver | Small result collection, testing | Returns list of Row objects | Dangerous on large data, OOM risk | `rows = df.collect()`
`for row in rows:`
` print(row.name)` | 21, 22 | | take() / head() | Retrieve first n rows | Sampling, preview, testing | Limits rows returned, safer than collect | Not random, takes first n | `rows = df.take(5)`
`first_row = df.head()` | 21, 22 | | first() | Get first row | Quick sample, validation | Simple, returns single Row | Only one row, no guarantee of representativeness | `first_row = df.first()`
`print(first_row.name)` | 21, 22 | | foreach() / foreachPartition() | Apply function to each row/partition | Side effects, writing to external systems | Distributed execution, partition-level optimization | No return value, for side effects only | `df.foreach(lambda row: print(row))`
`def process_partition(iter):`
` for row in iter: # write to DB`
`df.foreachPartition(process_partition)` | 21, 22 | | write() | Write DataFrame to storage | Saving results, ETL outputs, persistence | Multiple formats, modes, partitioning | Blocking operation, ensure path is correct | `df.write.parquet("output_path")`
`df.write.mode("overwrite").csv("output")` | 12, 13 | | cache() / persist() | Cache DataFrame in memory/disk | Reuse DataFrames, iterative algorithms, performance | Speeds up repeated access, configurable storage levels | Uses memory, manually uncache when done | `df.cache()`
`from pyspark import StorageLevel`
`df.persist(StorageLevel.MEMORY_AND_DISK)` | 31, 32, 33 | | unpersist() | Remove DataFrame from cache | Memory management, freeing resources | Explicit cache control | Must be called to free memory | `df.unpersist()` | 31, 32 | | checkpoint() | Save DataFrame to reliable storage for fault tolerance | Long lineages, iterative algorithms, fault tolerance | Breaks lineage, improves fault tolerance | Requires reliable storage (HDFS), overhead | `spark.sparkContext.setCheckpointDir("/checkpoint")`
`df = df.checkpoint()` | 31, 34 | | toPandas() | Convert to Pandas DataFrame | Pandas compatibility, plotting, local analysis | Use Pandas ecosystem, visualizations | Collects to driver, OOM risk on large data | `pandas_df = df.toPandas()`
`pandas_df.plot()` | 35, 36 | --- ## Table 5: Column Expressions and Functions | Technique/Concept | Description | Main Usages | Advantages | Considerations | Example | Reference | |---|---|---|---|---|---|---| | col() / column() | Reference column by name | Column operations, transformations | Type-safe, explicit, supports column methods | Requires import | `from pyspark.sql.functions import col`
`df.select(col("age") + 1)` | 23, 37 | | lit() | Create literal constant column | Adding constants, flags, default values | Type-safe constants, various types supported | Creates column object | `from pyspark.sql.functions import lit`
`df.withColumn("constant", lit(100))` | 23, 37 | | expr() | Parse SQL expression string | Complex SQL expressions, dynamic columns | SQL syntax, powerful expressions | String-based, less type-safe | `from pyspark.sql.functions import expr`
`df.withColumn("result", expr("age * 2 + 10"))` | 23, 24, 37 | | when() / otherwise() | Conditional expressions (like SQL CASE) | Conditional logic, categorization, flagging | Chainable, multiple conditions, readable | Requires otherwise() for else case | `from pyspark.sql.functions import when`
`df.withColumn("category",`
` when(col("age") < 18, "minor")`
` .when(col("age") < 65, "adult")`
` .otherwise("senior"))` | 38, 39 | | cast() / astype() | Convert column data type | Type conversion, schema alignment | Explicit type control, handles conversions | Can fail if incompatible, returns null on failure | `df.withColumn("age_str", col("age").cast("string"))`
`df.withColumn("age_int", col("age_str").cast(IntegerType()))` | 21, 22 | | alias() | Rename column | Column renaming, clarity, avoiding conflicts | Simple renaming | Only for display, doesn't change schema | `df.select(col("name").alias("full_name"))` | 21, 22 | | isNull() / isNotNull() | Check for null values | Data quality, filtering, validation | Boolean conditions, chainable | Must use for null checks, == None doesn't work | `df.filter(col("age").isNotNull())`
`df.filter(col("name").isNull())` | 40, 41 | | isin() | Check if value in list | Filtering, membership tests | Cleaner than multiple OR conditions | List must be provided | `df.filter(col("status").isin(["active", "pending"]))` | 21, 22 | | substr() / substring() | Extract substring | Text processing, truncation, extraction | Position-based extraction | 1-indexed, requires start and length | `from pyspark.sql.functions import substring`
`df.withColumn("short", substring("name", 1, 3))` | 37, 42 | | concat() / concat_ws() | Concatenate columns | String building, combining fields | Multiple columns, separator support | Null in any column makes result null (use coalesce) | `from pyspark.sql.functions import concat, concat_ws`
`df.withColumn("full", concat(col("first"), lit(" "), col("last")))`
`df.withColumn("full", concat_ws(" ", "first", "last"))` | 37, 42 | | upper() / lower() / trim() | String case and whitespace operations | Text normalization, cleaning, standardization | Common text operations, simple | Creates new column | `from pyspark.sql.functions import upper, lower, trim`
`df.withColumn("upper_name", upper("name"))` | 37, 42 | | split() | Split string into array | Tokenization, parsing delimited strings | Returns array, delimiter specified | Creates array column | `from pyspark.sql.functions import split`
`df.withColumn("name_parts", split("name", " "))` | 37, 42, 43 | --- ## Table 6: Aggregation and GroupBy Operations | Technique/Concept | Description | Main Usages | Advantages | Considerations | Example | Reference | |---|---|---|---|---|---|---| | groupBy() | Group rows by columns for aggregation | Aggregations, summaries, analysis | Multiple grouping columns, chainable aggregations | Creates RelationalGroupedDataset, requires agg | `df.groupBy("department").agg(sum("salary"))` | 44, 45 | | agg() | Apply aggregate functions | Aggregations after groupBy, multiple metrics | Multiple aggregations in one call, named columns | Requires groupBy first | `from pyspark.sql.functions import sum, avg, count`
`df.groupBy("dept").agg(`
` sum("salary").alias("total"),`
` avg("age").alias("avg_age"))` | 44, 45 | | count() / sum() / avg() / min() / max() | Basic aggregate functions | Statistical summaries, metrics | Simple, common operations | Can be slow on large data | `from pyspark.sql.functions import sum, avg`
`df.groupBy("dept").agg(sum("salary"), avg("age"))` | 37, 44, 45 | | countDistinct() / approx_count_distinct() | Count distinct values | Cardinality, unique counts | Exact vs approximate (faster) | approx uses HyperLogLog, trading accuracy for speed | `from pyspark.sql.functions import countDistinct, approx_count_distinct`
`df.agg(countDistinct("user_id"))`
`df.agg(approx_count_distinct("user_id", 0.05))` | 37, 44 | | collect_list() / collect_set() | Collect values into array | Aggregating arrays, grouping values | List preserves duplicates, set removes them | Can create large arrays, memory intensive | `from pyspark.sql.functions import collect_list, collect_set`
`df.groupBy("user").agg(collect_list("item"))` | 37, 43, 44 | | first() / last() | Get first/last value in group | Sampling, representative values | Simple extraction | Order matters, may need orderBy first | `from pyspark.sql.functions import first, last`
`df.groupBy("id").agg(first("timestamp"))` | 37, 44 | | pivot() | Rotate rows to columns | Cross-tabulation, wide format transformation | Creates matrix-like structure | Can create many columns, memory intensive | `df.groupBy("year").pivot("quarter").agg(sum("sales"))` | 46, 47 | | unpivot() / melt() | Rotate columns to rows | Long format transformation, normalizing data | Reverse of pivot, standard format for analysis | Requires identifying id and value columns | `df.unpivot("id", ["col1", "col2"], "variable", "value")`
`# Or use melt()`
`df.melt(ids=["id"], values=["col1", "col2"])` | 46, 47, 48 | | rollup() | Multi-level aggregation with subtotals | Hierarchical aggregations, reporting | Produces subtotals at all levels | Creates null for subtotal rows | `df.rollup("country", "state").agg(sum("sales"))` | 44, 45 | | cube() | Multi-dimensional aggregation | OLAP-style aggregations, data cubes | All combinations of grouping columns | Can explode result size with many columns | `df.cube("year", "quarter", "region").agg(sum("sales"))` | 44, 45 | --- ## Table 7: Window Functions | Technique/Concept | Description | Main Usages | Advantages | Considerations | Example | Reference | |---|---|---|---|---|---|---| | Window.partitionBy() | Define window partitions | Partition data for window functions | Groups data for ranking/aggregation within groups | Requires Window import | `from pyspark.sql import Window`
`window = Window.partitionBy("department")` | 49, 50, 51 | | Window.orderBy() | Define ordering within windows | Ranking, cumulative operations, row numbering | Specifies order for row-dependent operations | Can be ascending or descending | `window = Window.partitionBy("dept").orderBy("salary")` | 49, 50, 51 | | Window.rowsBetween() | Define row-based frame | Moving averages, cumulative sums, sliding windows | Precise row control, physical rows | Row-based, not value-based | `from pyspark.sql import Window`
`from pyspark.sql.functions import avg`
`window = Window.orderBy("date").rowsBetween(-2, 0)`
`df.withColumn("moving_avg", avg("value").over(window))` | 49, 50, 51 | | Window.rangeBetween() | Define value-based frame | Time windows, value-based ranges | Value-based, not row-based | Requires ordered column | `window = Window.orderBy("timestamp").rangeBetween(-3600, 0)` | 49, 50, 51 | | row_number() | Assign unique number to rows | Ranking, deduplication, pagination | Unique numbers, deterministic with orderBy | Requires orderBy for determinism | `from pyspark.sql.functions import row_number`
`window = Window.partitionBy("dept").orderBy("salary")`
`df.withColumn("rank", row_number().over(window))` | 49, 50, 51, 52 | | rank() / dense_rank() | Assign ranks with ties | Ranking with gaps (rank) or without (dense_rank) | Handles ties, rank vs dense_rank behavior | Rank skips numbers after ties, dense_rank doesn't | `from pyspark.sql.functions import rank, dense_rank`
`df.withColumn("rank", rank().over(window))`
`df.withColumn("dense", dense_rank().over(window))` | 49, 50, 51, 52 | | lag() / lead() | Access previous/next row values | Time series, comparing to previous values | Offset-based access, default values | Requires orderBy | `from pyspark.sql.functions import lag, lead`
`df.withColumn("prev", lag("value", 1).over(window))`
`df.withColumn("next", lead("value", 1, 0).over(window))` | 49, 50, 51, 52 | | ntile() | Divide into n buckets | Percentiles, quartiles, bucketing | Equal-sized groups | Requires orderBy | `from pyspark.sql.functions import ntile`
`df.withColumn("quartile", ntile(4).over(window))` | 49, 50, 51, 52 | | Window Aggregations | Use agg functions with windows | Running totals, moving averages, cumulative metrics | Aggregations within partitions | All agg functions work with windows | `from pyspark.sql.functions import sum, avg`
`df.withColumn("running_total", sum("amount").over(window))`
`df.withColumn("moving_avg", avg("value").over(window))` | 49, 50, 51 | --- ## Table 8: Null Handling and Data Quality | Technique/Concept | Description | Main Usages | Advantages | Considerations | Example | Reference | |---|---|---|---|---|---|---| | fillna() / fill() | Replace null values | Data cleaning, imputation, default values | Multiple strategies, column-specific | Doesn't modify original, returns new DataFrame | `df.fillna(0)`
`df.fillna({"age": 0, "name": "Unknown"})`
`df.na.fill({"salary": df.agg({"salary": "avg"}).collect()[0][0]})` | 40, 41, 53 | | dropna() / drop() | Remove rows with nulls | Data cleaning, removing incomplete records | Threshold parameter (any/all), column subset | Loses data, may reduce dataset significantly | `df.dropna()`
`df.dropna(how="all")`
`df.dropna(subset=["name", "age"])`
`df.dropna(thresh=2)` | 40, 41, 53 | | replace() | Replace specific values | Value standardization, cleaning | Multiple values, dictionary mapping | Only replaces exact matches | `df.replace(["NA", "N/A"], None)`
`df.replace({0: None, -1: None})` | 40, 53 | | coalesce() | Return first non-null value | Default values, null handling in expressions | Multiple fallback columns | Returns first non-null, order matters | `from pyspark.sql.functions import coalesce`
`df.withColumn("result", coalesce("col1", "col2", lit("default")))` | 40, 41, 53 | | when().isNull() | Conditional null handling | Custom null logic, conditional replacement | Flexible conditions | More verbose than fillna for simple cases | `from pyspark.sql.functions import when`
`df.withColumn("age_clean",`
` when(col("age").isNull(), 0).otherwise(col("age")))` | 38, 40, 41 | | na methods (DataFrameNaFunctions) | Specialized null handling operations | Advanced null handling, statistics | Dedicated API for null operations | Accessed via df.na | `df.na.fill(0)`
`df.na.drop("any")`
`df.na.replace({"old": "new"})` | 40, 41, 53 | --- ## Table 9: Array and Collection Functions | Technique/Concept | Description | Main Usages | Advantages | Considerations | Example | Reference | |---|---|---|---|---|---|---| | explode() | Expand array/map to multiple rows | Unnesting arrays, flattening data | Simple expansion, one row per element | Null arrays create no rows (use explode_outer) | `from pyspark.sql.functions import explode`
`df.select("id", explode("items").alias("item"))` | 43, 54, 55 | | explode_outer() | Explode with null preservation | Unnesting with null handling | Preserves rows with null arrays | Creates row with null if array is null | `from pyspark.sql.functions import explode_outer`
`df.select("id", explode_outer("items").alias("item"))` | 43, 54 | | collect_list() / collect_set() | Aggregate into arrays | Grouping items, creating lists | List keeps duplicates, set removes them | Can create large arrays | `from pyspark.sql.functions import collect_list`
`df.groupBy("user").agg(collect_list("item").alias("items"))` | 37, 43, 44 | | array_contains() | Check if array contains value | Filtering, searching arrays | Boolean condition for filtering | Null-safe | `from pyspark.sql.functions import array_contains`
`df.filter(array_contains("items", "apple"))` | 37, 43, 54 | | array() | Create array from columns | Combining columns, array creation | Multiple columns to array | Null handling important | `from pyspark.sql.functions import array`
`df.withColumn("combined", array("col1", "col2", "col3"))` | 37, 43 | | size() / cardinality() | Get array/map size | Array length, counting elements | Simple size calculation | Returns -1 for null (not 0) | `from pyspark.sql.functions import size`
`df.withColumn("array_len", size("items"))` | 37, 43, 54 | | array_distinct() | Remove duplicates from array | Deduplication, array cleaning | Preserves array structure | Null handling | `from pyspark.sql.functions import array_distinct`
`df.withColumn("unique", array_distinct("items"))` | 37, 43, 54 | | array_sort() / sort_array() | Sort array elements | Ordering, normalization | Ascending/descending | Modifies order | `from pyspark.sql.functions import sort_array`
`df.withColumn("sorted", sort_array("items"))` | 37, 43, 54 | | slice() | Extract array subset | Array slicing, sampling | Position-based extraction | 1-indexed in Spark | `from pyspark.sql.functions import slice`
`df.withColumn("first_3", slice("items", 1, 3))` | 37, 43, 54 | | array_join() | Join array elements to string | Converting arrays to strings, concatenation | Delimiter specified | Null handling | `from pyspark.sql.functions import array_join`
`df.withColumn("joined", array_join("items", ","))` | 37, 43 | | flatten() | Flatten nested arrays | Multi-level array flattening | One level flattening | Only one level deep | `from pyspark.sql.functions import flatten`
`df.withColumn("flat", flatten("nested_arrays"))` | 37, 43, 54 | --- ## Table 10: User-Defined Functions (UDFs) | Technique/Concept | Description | Main Usages | Advantages | Considerations | Example | Reference | |---|---|---|---|---|---|---| | Standard UDF (@udf) | Python function applied row-by-row | Custom transformations, complex logic | Flexible, any Python code | Slow (serialization overhead), no optimization | `from pyspark.sql.functions import udf`
`from pyspark.sql.types import IntegerType`
`@udf(returnType=IntegerType())`
`def square(x):`
` return x * x`
`df.withColumn("squared", square("value"))` | 56, 57, 58 | | Pandas UDF (Vectorized) | Vectorized function on Pandas Series/DataFrame | Performance-critical transformations, batch processing | Much faster than standard UDF, vectorized operations | Requires pandas, arrow, specific type signatures | `from pyspark.sql.functions import pandas_udf`
`import pandas as pd`
`@pandas_udf("int")`
`def square(s: pd.Series) -> pd.Series:`
` return s * s`
`df.withColumn("squared", square("value"))` | 56, 57, 59, 60 | | Scalar Pandas UDF | Pandas UDF for scalar operations | Element-wise transformations | Vectorized, faster than standard UDF | Input/output are Series | `@pandas_udf("double")`
`def multiply(a: pd.Series, b: pd.Series) -> pd.Series:`
` return a * b` | 56, 57, 59 | | Grouped Map Pandas UDF | Apply function to each group | Group-wise operations, per-group models | Full pandas DataFrame per group, flexible | Entire group in memory, can be large | `from pyspark.sql.functions import pandas_udf, PandasUDFType`
`@pandas_udf(schema, PandasUDFType.GROUPED_MAP)`
`def process_group(pdf):`
` # process pandas df`
` return pdf`
`df.groupBy("group").apply(process_group)` | 56, 57, 59, 60 | | Pandas Function API | Transform entire DataFrame with pandas | DataFrame-level operations, applyInPandas | Flexible, entire DF or groups | Memory considerations | `def transform(pdf):`
` # pandas operations`
` return pdf`
`df.groupBy("key").applyInPandas(transform, schema)` | 56, 57, 59 | | Return Type Specification | Define UDF return type | Type safety, optimization | Clear schema, enables optimization | Must match actual return type | `from pyspark.sql.types import StructType, StructField, IntegerType`
`schema = StructType([StructField("result", IntegerType())])`
`@udf(returnType=schema)` | 56, 57, 58 | --- ## Table 11: Performance Optimization | Technique/Concept | Description | Main Usages | Advantages | Considerations | Example | Reference | |---|---|---|---|---|---|---| | Broadcast Join | Broadcast small table to all executors | Joining small dimension tables | No shuffle for small table, very fast | Small table must fit in memory (<10MB default) | `from pyspark.sql.functions import broadcast`
`large_df.join(broadcast(small_df), "key")` | 28, 29, 30, 61 | | Partitioning | Control data distribution across cluster | Optimizing joins, reducing shuffles, parallelism | Matched partition keys avoid shuffles | Repartition is expensive, choose wisely | `df.repartition(200, "key")`
`df.coalesce(10) # reduce partitions` | 9, 62, 63 | | repartition() | Increase or decrease partitions with full shuffle | Rebalancing data, increasing parallelism | Full shuffle, any number of partitions | Expensive operation, triggers shuffle | `df.repartition(100)`
`df.repartition("key") # hash partitioning` | 9, 62, 63 | | coalesce() | Reduce partitions without full shuffle | Reducing partitions, before writing | No full shuffle, cheaper than repartition | Only reduces, cannot increase partitions | `df.coalesce(10)` | 9, 62, 63 | | Caching / Persistence | Store DataFrame in memory/disk | Reusing DataFrames, iterative algorithms | Speeds up repeated access | Uses memory, remember to unpersist | `df.cache() # MEMORY_ONLY`
`df.persist(StorageLevel.MEMORY_AND_DISK)` | 31, 32, 33 | | Predicate Pushdown | Push filters to data source | Reading less data, optimization | Automatic, reduces data scanned | Works with Parquet, ORC, JDBC | `# Automatic when reading`
`df = spark.read.parquet("data")`
`df.filter("date > '2024-01-01'")` | 64, 65 | | Column Pruning | Select only needed columns | Reducing data read, memory usage | Automatic with columnar formats | Works with Parquet, ORC | `# Only reads needed columns`
`df.select("col1", "col2").show()` | 64, 65 | | Bucketing | Pre-partition data by hash | Optimizing joins, reducing shuffles | Avoids shuffle for joins on bucketed columns | Must write bucketed, requires Hive support | `df.write.bucketBy(100, "key").sortBy("value").saveAsTable("bucketed_table")` | 30, 62, 66 | | Catalyst Optimizer | Query optimizer (logical/physical plan) | Automatic optimization, query rewriting | Automatic, rule-based, cost-based | Transparent to user, trust but verify with explain() | `# Automatic, view with:`
`df.explain(extended=True)` | 67, 68, 69 | | Tungsten Engine | Memory/CPU optimization engine | Code generation, memory management | Faster execution, off-heap memory | Automatic, binary processing | `# Automatic, enabled by default` | 67, 68, 70 | | Adaptive Query Execution (AQE) | Runtime query optimization | Dynamic optimization during execution | Optimizes based on runtime statistics | Enable with config | `spark.conf.set("spark.sql.adaptive.enabled", "true")`
`spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "true")` | 71, 72 | | Avoid UDFs When Possible | Use built-in functions instead of UDFs | Performance optimization | Built-in functions optimized, no serialization | UDFs bypass Catalyst optimization | `# Avoid:`
`@udf`
`def upper(s): return s.upper()`
`# Use:`
`from pyspark.sql.functions import upper`
`df.withColumn("upper", upper("name"))` | 56, 61 | | Salting | Add random key to distribute skewed data | Handling data skew in joins | Distributes hot keys across partitions | Requires additional column, manual implementation | `import random`
`df = df.withColumn("salt", lit(random.randint(0, 10)))`
`df = df.repartition("key", "salt")` | 61, 73 | --- ## Table 12: RDD Operations (Low-Level) | Technique/Concept | Description | Main Usages | Advantages | Considerations | Example | Reference | |---|---|---|---|---|---|---| | RDD Creation | Create Resilient Distributed Dataset | Low-level operations, unstructured data | Fine-grained control, transformations | Lower-level than DataFrame, less optimization | `rdd = spark.sparkContext.parallelize([1, 2, 3, 4])`
`rdd = spark.sparkContext.textFile("file.txt")` | 74, 75 | | map() | Transform each element | Element-wise transformation | Simple transformation | One-to-one mapping | `rdd = rdd.map(lambda x: x * 2)` | 74, 75, 76 | | filter() | Filter elements | Selecting subset | Simple filtering | Returns elements matching predicate | `rdd = rdd.filter(lambda x: x > 5)` | 74, 75, 76 | | flatMap() | Map and flatten | One-to-many transformation | Flexible output | Each element can produce multiple outputs | `rdd = rdd.flatMap(lambda x: [x, x*2])` | 74, 75, 76 | | reduce() / reduceByKey() | Aggregate elements | Aggregation, summation | Reduce-side aggregation, efficient for key-value | Requires associative, commutative function | `total = rdd.reduce(lambda a, b: a + b)`
`rdd.reduceByKey(lambda a, b: a + b)` | 74, 75, 76 | | groupByKey() | Group values by key | Grouping data | Simple grouping | Shuffles all data, less efficient than reduceByKey | `grouped = rdd.groupByKey()` | 74, 75, 76 | | sortBy() / sortByKey() | Sort RDD | Ordering data | Distributed sort | Expensive, requires shuffle | `sorted_rdd = rdd.sortBy(lambda x: x[1])` | 74, 75, 76 | | Transformations vs Actions | Lazy vs eager operations | Understanding Spark execution model | Transformations lazy (map, filter), actions eager (collect, count) | Only actions trigger execution | `# Transformation (lazy):`
`rdd2 = rdd.map(lambda x: x*2)`
`# Action (eager):`
`result = rdd2.collect()` | 74, 75, 77 | | toDF() | Convert RDD to DataFrame | Upgrading to DataFrame API | Structured data, SQL operations | Requires schema definition or inference | `df = rdd.toDF(["col1", "col2"])`
`df = rdd.toDF()` | 74, 75 | --- ## Table 13: Structured Streaming | Technique/Concept | Description | Main Usages | Advantages | Considerations | Example | Reference | |---|---|---|---|---|---|---| | readStream | Create streaming DataFrame | Real-time data ingestion, streaming ETL | Continuous processing, incremental | Requires streaming source (Kafka, socket, files) | `streaming_df = spark.readStream`
` .format("kafka")`
` .option("kafka.bootstrap.servers", "host:9092")`
` .option("subscribe", "topic")`
` .load()` | 78, 79, 80 | | writeStream | Write streaming DataFrame | Output streaming results, sinks | Multiple output modes, fault tolerance | Requires output sink and trigger | `query = streaming_df.writeStream`
` .format("parquet")`
` .option("path", "output")`
` .option("checkpointLocation", "checkpoint")`
` .start()` | 78, 79, 80 | | Output Modes | Append, Complete, Update | Control output behavior | Append=new rows, Complete=entire table, Update=changed rows | Mode depends on query type | `query = streaming_df.writeStream`
` .outputMode("append")`
` .format("console")`
` .start()` | 78, 79, 80 | | Watermarking | Handle late data in streaming | Event-time processing, windowed aggregations | Defines how late data is handled | Must specify watermark column and threshold | `from pyspark.sql.functions import window`
`windowed = streaming_df`
` .withWatermark("timestamp", "10 minutes")`
` .groupBy(window("timestamp", "5 minutes"))`
` .count()` | 78, 81, 82, 83 | | Window Operations | Time-based aggregations in streaming | Tumbling, sliding, session windows | Event-time windowing, aggregations | Requires timestamp column | `from pyspark.sql.functions import window`
`windowed = streaming_df.groupBy(`
` window("timestamp", "10 minutes", "5 minutes")`
`).count()` | 78, 81, 82 | | Triggers | Control micro-batch frequency | Processing time control, latency tuning | Continuous, processing time, once, available now | Default=as fast as possible | `query = streaming_df.writeStream`
` .trigger(processingTime="10 seconds")`
` .start()`
`query = streaming_df.writeStream`
` .trigger(once=True)`
` .start()` | 78, 79, 80 | | Checkpointing | Fault tolerance for streaming | Recovery, exactly-once semantics | Enables recovery after failures | Requires reliable storage (HDFS, S3) | `query = streaming_df.writeStream`
` .option("checkpointLocation", "/checkpoint")`
` .start()` | 78, 79, 80 | | Stream-Stream Joins | Join two streaming DataFrames | Correlating events from multiple streams | Event matching, enrichment | Requires watermarks on both streams | `joined = stream1`
` .withWatermark("timestamp1", "10 minutes")`
` .join(stream2.withWatermark("timestamp2", "10 minutes"),`
` expr("timestamp1 >= timestamp2 AND timestamp1 <= timestamp2 + interval 5 minutes"))`
`` | 78, 84 | | Stream-Static Joins | Join streaming with static DataFrame | Enrichment, dimension lookups | Simple enrichment, no watermark needed | Static side must fit in memory or be broadcast | `enriched = streaming_df.join(static_df, "key")` | 78, 84 | --- ## Table 14: Machine Learning (MLlib) | Technique/Concept | Description | Main Usages | Advantages | Considerations | Example | Reference | |---|---|---|---|---|---|---| | VectorAssembler | Combine features into vector column | Feature preparation for ML algorithms | Required for most ML algorithms | Creates single vector column | `from pyspark.ml.feature import VectorAssembler`
`assembler = VectorAssembler(`
` inputCols=["age", "salary"],`
` outputCol="features")`
`df_assembled = assembler.transform(df)` | 85, 86, 87 | | StringIndexer | Convert string labels to indices | Encoding categorical features | Handles categorical features | Creates integer indices | `from pyspark.ml.feature import StringIndexer`
`indexer = StringIndexer(inputCol="category", outputCol="categoryIndex")`
`indexed = indexer.fit(df).transform(df)` | 85, 86, 87 | | OneHotEncoder | One-hot encode categorical features | Categorical encoding for ML | Standard encoding method | Creates sparse vectors | `from pyspark.ml.feature import OneHotEncoder`
`encoder = OneHotEncoder(inputCol="categoryIndex", outputCol="categoryVec")`
`encoded = encoder.fit(df).transform(df)` | 85, 86, 87 | | StandardScaler / MinMaxScaler | Scale/normalize features | Feature normalization, improving convergence | Common preprocessing step | May not be needed for tree-based models | `from pyspark.ml.feature import StandardScaler`
`scaler = StandardScaler(inputCol="features", outputCol="scaledFeatures")`
`scaled = scaler.fit(df).transform(df)` | 85, 86, 87 | | Train-Test Split | Split data for training and testing | Model evaluation, validation | Simple splitting, stratification support | Random split, set seed for reproducibility | `train_df, test_df = df.randomSplit([0.8, 0.2], seed=42)` | 85, 86, 87 | | LogisticRegression | Binary/multiclass classification | Classification problems | Interpretable, probabilistic | Assumes linear decision boundary | `from pyspark.ml.classification import LogisticRegression`
`lr = LogisticRegression(featuresCol="features", labelCol="label")`
`model = lr.fit(train_df)`
`predictions = model.transform(test_df)` | 85, 86, 88 | | RandomForestClassifier | Ensemble tree-based classification | Classification with complex patterns | Handles non-linear relationships, feature importance | More parameters to tune | `from pyspark.ml.classification import RandomForestClassifier`
`rf = RandomForestClassifier(numTrees=100)`
`model = rf.fit(train_df)` | 85, 86, 88 | | LinearRegression | Linear regression modeling | Continuous target prediction | Simple, interpretable | Assumes linear relationship | `from pyspark.ml.regression import LinearRegression`
`lr = LinearRegression(featuresCol="features", labelCol="target")`
`model = lr.fit(train_df)` | 85, 86, 88 | | KMeans | Clustering algorithm | Unsupervised grouping, customer segmentation | Simple, scalable | Requires specifying k | `from pyspark.ml.clustering import KMeans`
`kmeans = KMeans(k=3, seed=1)`
`model = kmeans.fit(df)`
`predictions = model.transform(df)` | 85, 86, 89 | | Pipeline | Chain multiple stages | Workflow automation, reproducibility | Encapsulates entire workflow | Can be complex for debugging | `from pyspark.ml import Pipeline`
`pipeline = Pipeline(stages=[indexer, assembler, scaler, lr])`
`model = pipeline.fit(train_df)`
`predictions = model.transform(test_df)` | 85, 86, 87 | | CrossValidator | Cross-validation for hyperparameter tuning | Model selection, hyperparameter tuning | Systematic parameter search | Computationally expensive | `from pyspark.ml.tuning import CrossValidator, ParamGridBuilder`
`from pyspark.ml.evaluation import BinaryClassificationEvaluator`
`paramGrid = ParamGridBuilder()`
` .addGrid(lr.regParam, [0.01, 0.1])`
` .build()`
`cv = CrossValidator(estimator=lr, estimatorParamMaps=paramGrid, evaluator=BinaryClassificationEvaluator())`
`cvModel = cv.fit(train_df)` | 85, 86, 87 | | Model Evaluation | Evaluate model performance | Model assessment, metrics | Multiple evaluators available | Choose appropriate metric for problem | `from pyspark.ml.evaluation import MulticlassClassificationEvaluator`
`evaluator = MulticlassClassificationEvaluator(metricName="accuracy")`
`accuracy = evaluator.evaluate(predictions)` | 85, 86, 87 | --- ## Table 15: Advanced Topics and Best Practices | Technique/Concept | Description | Main Usages | Advantages | Considerations | Example | Reference | |---|---|---|---|---|---|---| | Lazy Evaluation | Transformations not executed until action | Performance, optimization, plan building | Allows Catalyst optimization, reduces unnecessary computation | Must call action to execute | `# Transformations are lazy:`
`df2 = df.filter("age > 25") # not executed`
`df3 = df2.select("name") # not executed`
`df3.show() # now executed` | 74, 77, 90 | | DAG (Directed Acyclic Graph) | Execution plan visualization | Understanding execution, debugging | Shows stages, dependencies, optimizations | View with Spark UI | `# View DAG in Spark UI`
`df.explain(extended=True) # text plan` | 67, 91 | | Spark UI | Web interface for monitoring | Debugging, performance tuning, monitoring | Detailed metrics, stages, tasks, storage | Usually http://localhost:4040 | `# Access at http://driver:4040`
`# View stages, executors, storage, SQL` | 92, 93 | | explain() | Show query execution plan | Understanding optimization, debugging | Shows logical and physical plans | extended=True for full detail | `df.explain()`
`df.explain(extended=True)`
`df.explain("formatted")` | 67, 94 | | Schema Merging | Merge schemas when reading files | Handling schema evolution, multiple sources | Automatic schema discovery | Can be slow, explicit schema better | `df = spark.read.option("mergeSchema", "true").parquet("data")` | 18, 95 | | Dynamic Partition Overwrite | Overwrite only affected partitions | Incremental updates, partition management | Only overwrites specified partitions | Requires partitioning | `spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic")`
`df.write.mode("overwrite").partitionBy("date").parquet("output")` | 96, 97 | | Skew Handling | Handle data skew in joins/aggregations | Fixing performance bottlenecks, long tasks | Salting, broadcast, skew join optimization | Manual or automatic (AQE) | `# Enable skew join optimization (AQE)`
`spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true")` | 61, 71, 73 | | Data Skew Detection | Identify skewed partitions | Performance diagnosis | Spark UI shows task times | Look for outlier tasks | `# Check partition sizes`
`df.rdd.glom().map(len).collect()` | 61, 73 | | Memory Tuning | Optimize memory configuration | Preventing OOM, performance | Executor, driver, overhead memory | Balance storage, execution, overhead | `spark.conf.set("spark.executor.memory", "8g")`
`spark.conf.set("spark.memory.fraction", "0.8")`
`spark.conf.set("spark.memory.storageFraction", "0.3")` | 5, 98 | | Serialization (Kryo) | Faster serialization than Java default | Performance optimization | Faster, more compact | Requires registration for custom classes | `spark.conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")` | 99, 100 | | SQL Temp Views | Register DataFrame as SQL table | SQL queries on DataFrames | Familiar SQL syntax | Scoped to session, not persisted | `df.createOrReplaceTempView("people")`
`spark.sql("SELECT * FROM people WHERE age > 25").show()` | 21, 101 | | Global Temp Views | Cross-session temporary views | Sharing data across sessions | Global scope | Requires global_temp prefix | `df.createOrReplaceGlobalTempView("people")`
`spark.sql("SELECT * FROM global_temp.people").show()` | 21, 101 | | Repartition vs Coalesce | When to use each | Optimization, partition management | Repartition=full shuffle, Coalesce=no shuffle (reduce only) | Use coalesce to reduce, repartition to increase or rebalance | `# Reduce partitions (no shuffle):`
`df.coalesce(10)`
`# Increase or rebalance (full shuffle):`
`df.repartition(100)` | 9, 62, 63 | | BroadcastTimeout | Timeout for broadcast joins | Preventing hangs, large broadcast handling | Configurable timeout | Increase if broadcast takes time | `spark.conf.set("spark.sql.broadcastTimeout", 600)` | 30, 102 | --- ## References ### Official Apache Spark Documentation 1. SparkSession - https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/spark_session.html 2. PySpark Getting Started - https://spark.apache.org/docs/latest/api/python/getting_started/index.html 3. Spark Configuration - https://spark.apache.org/docs/latest/configuration.html 4. Spark Configuration Guide - https://spark.apache.org/docs/latest/configuration.html#application-properties 5. Executor Memory Configuration - https://sparkbyexamples.com/spark/how-to-set-apache-spark-executor-memory/ 6. Spark Cluster Mode Overview - https://spark.apache.org/docs/latest/cluster-overview.html 7. Spark Deployment Modes - https://www.informit.com/articles/article.aspx?p=2928186&seqNum=3 8. Spark Execution Modes - https://dataloader.substack.com/p/spark-execution-modes-cluster-client ### DataFrame Operations and APIs 10. PySpark DataFrame - https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/dataframe.html 11. DataFrame Quickstart - https://spark.apache.org/docs/latest/api/python/getting_started/quickstart_df.html 12. Data Sources - https://spark.apache.org/docs/latest/sql-data-sources-load-save-functions.html 13. File Formats - https://medium.com/@dhanashrisaner.30/file-formats-and-data-sources-in-pyspark-14de6a029b74 14. Reading CSV - https://builtin.com/articles/spark-read-csv 15. Parquet Files - https://spark.apache.org/docs/latest/sql-data-sources-parquet.html 16. JSON Files - https://spark.apache.org/docs/latest/sql-data-sources-json.html 17. JDBC Data Sources - https://spark.apache.org/docs/latest/sql-data-sources-jdbc.html 18. StructType and StructField - https://sparkbyexamples.com/pyspark/pyspark-structtype-and-structfield/ 19. Schema Definition - https://www.mungingdata.com/pyspark/schema-structtype-structfield/ 20. PySpark StructType - https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.types.StructType.html 21. DataFrame API Reference - https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/dataframe.html 22. PySpark DataFrame Tutorial - https://sparkbyexamples.com/pyspark-dataframe-tutorial-with-examples/ 23. PySpark expr() Function - https://sparkbyexamples.com/pyspark/pyspark-sql-expr-expression-function/ 24. selectExpr() Function - https://sparkbyexamples.com/pyspark/pyspark-selectexpr/ 25. withColumn() Examples - https://sparkbyexamples.com/pyspark/pyspark-withcolumn/ 26. union() and unionByName() - https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.union.html ### Joins and Performance 27. PySpark Joins - https://www.datacamp.com/tutorial/pyspark-joins 28. Join Operations - https://sparkbyexamples.com/pyspark/pyspark-join-explained-with-examples/ 29. Broadcast Joins - https://www.sparkcodehub.com/pyspark/performance/broadcast-joins 30. Broadcast Join Optimization - https://towardsdatascience.com/apache-spark-performance-boosting-e072a3ec1179 31. Cache and Persist - https://sparkbyexamples.com/pyspark/pyspark-cache-and-persist-explained/ 32. PySpark Caching - https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.cache.html 33. Storage Levels - https://spark.apache.org/docs/latest/rdd-programming-guide.html#rdd-persistence 34. Checkpointing - https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.checkpoint.html ### Pandas Integration and Conversion 35. toPandas() - https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.toPandas.html 36. Pandas API on Spark - https://spark.apache.org/docs/latest/api/python/user_guide/pandas_on_spark/index.html 37. PySpark Functions - https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/functions.html ### Conditional Logic and Expressions 38. when() and otherwise() - https://sparkbyexamples.com/pyspark/pyspark-when-otherwise/ 39. Case When Usage - https://pyspark.itversity.com/04_processing_column_data/19_using_case_and_when.html 40. Null Handling - https://medium.com/@omkarspatil2611/dealing-with-null-values-in-apache-spark-a3601d409240 41. PySpark Null Operations - https://www.sparkcodehub.com/pyspark-null-operations 42. String Functions - https://sparkbyexamples.com/pyspark/pyspark-sql-string-functions/ 43. Array Functions - https://sparkbyexamples.com/pyspark/pyspark-array-column-functions/ ### Aggregations and Grouping 44. groupBy() Operations - https://www.datacamp.com/tutorial/pyspark-groupby 45. PySpark Aggregations - https://sparkbyexamples.com/pyspark/pyspark-aggregate-functions/ 46. Pivot and Unpivot - https://sparkbyexamples.com/pyspark/pyspark-pivot-and-unpivot-dataframe/ 47. Pivot Operations - https://medium.com/@imtheguna/pyspark-pivot-unpivot-b173a23d2352 48. melt() Function - https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.melt.html ### Window Functions 49. Window Functions - https://sparkbyexamples.com/pyspark/pyspark-window-functions/ 50. PySpark Window - https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.Window.html 51. Window Operations Guide - https://medium.com/@daveshpandey/pyspark-window-functions-a-practical-guide-for-cleaner-aggregations-50eae3a325ba 52. Window Functions Documentation - https://spark.apache.org/docs/latest/sql-ref-syntax-qry-select-window.html 53. fillna() and dropna() - https://sparkbyexamples.com/pyspark/pyspark-fillna-fill-replace-null-values/ ### Array and Collection Operations 54. explode() Function - https://sparkbyexamples.com/pyspark/pyspark-explode-array-and-map-columns-to-rows/ 55. Explode and Collect - https://towardsdatascience.com/pyspark-explained-the-explode-and-collect-list-functions-834f45ff5ac5 56. UDF Guide - https://spark.apache.org/docs/latest/api/python/user_guide/udfandudtf.html 57. PySpark UDFs - https://towardsdatascience.com/pyspark-explained-user-defined-functions-40a71f924b1a 58. Standard UDFs - https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.udf.html 59. Pandas UDFs - https://docs.databricks.com/aws/en/udf/pandas.html 60. Vectorized UDFs - https://www.databricks.com/blog/2017/10/30/introducing-vectorized-udfs-for-pyspark.html ### Performance Optimization 61. PySpark Performance Tuning - https://medium.com/@gankur277/pyspark-optimization-and-performance-tuning-master-partitioning-caching-and-catalyst-part-4-97713e8ac16d 62. Partitioning Best Practices - https://medium.com/@zaynhere2025/pyspark-tricks-every-data-engineer-should-know-608047e57e14 63. Repartition vs Coalesce - https://sparkbyexamples.com/pyspark/pyspark-repartition-vs-coalesce/ 64. Predicate Pushdown - https://spark.apache.org/docs/latest/sql-performance-tuning.html 65. Column Pruning - https://spark.apache.org/docs/latest/sql-performance-tuning.html 66. Bucketing - https://www.startdataengineering.com/post/how-to-optimize-your-spark-jobs/ 67. Catalyst Optimizer - https://stackoverflow.com/questions/56197121/what-is-catalyst-optimizer-in-apache-spark-sql 68. Catalyst and Tungsten - https://medium.com/@ghoshsiddharth25/spark-interview-series-catalyst-optimizer-and-tungsten-engine-c94cfc8cb41d 69. Catalyst Optimizer Deep Dive - https://www.linkedin.com/pulse/catalyst-tungsten-apache-sparks-speeding-engine-deepak-rajak 70. Tungsten Engine - https://www.sparkcodehub.com/spark/performance/tungsten-optimization 71. Adaptive Query Execution - https://sparkbyexamples.com/spark/spark-adaptive-query-execution/ 72. AQE Guide - https://spark.apache.org/docs/latest/sql-performance-tuning.html#adaptive-query-execution 73. Data Skew Handling - https://medium.com/@vtrkayalrajan/spark-optimization-for-a-large-datasets-8a045986660e ### RDD Operations 74. RDD Programming Guide - https://spark.apache.org/docs/latest/rdd-programming-guide.html 75. PySpark RDD - https://spark.apache.org/docs/latest/api/python/reference/pyspark.core/rdd.html 76. RDD Transformations - https://sparkbyexamples.com/pyspark/pyspark-rdd-transformations/ 77. Lazy Evaluation - https://medium.com/@think-data/mastering-lazy-evaluation-a-must-know-for-pyspark-pros-ac855202495e ### Structured Streaming 78. Structured Streaming - https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html 79. Streaming Guide - https://www.sparkcodehub.com/pyspark/streaming/structured-streaming-overview 80. Streaming Operations - https://spark.apache.org/docs/latest/api/python/reference/pyspark.ss/index.html 81. Watermarking - https://www.sparkcodehub.com/pyspark/streaming/watermarking 82. Event-time Aggregation - https://www.databricks.com/blog/2017/05/08/event-time-aggregation-watermarking-apache-sparks-structured-streaming.html 83. Watermarking Deep Dive - https://www.databricks.com/blog/feature-deep-dive-watermarking-apache-spark-structured-streaming 84. Stream Joins - https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#stream-stream-joins ### Machine Learning (MLlib) 85. MLlib Guide - https://spark.apache.org/docs/latest/ml-guide.html 86. MLlib API Reference - https://spark.apache.org/docs/latest/api/python/reference/pyspark.ml.html 87. ML Pipelines - https://spark.apache.org/docs/latest/ml-pipeline.html 88. Classification and Regression - https://spark.apache.org/docs/latest/ml-classification-regression.html 89. Clustering - https://spark.apache.org/docs/latest/ml-clustering.html 90. Lazy Evaluation Benefits - https://towardsdatascience.com/3-reasons-why-sparks-lazy-evaluation-is-useful-ed06e27360c4 ### Advanced Topics 91. DAG Execution - https://www.sparkcodehub.com/spark/fundamentals/how-dag-execution-plans-work 92. Spark UI - https://spark.apache.org/docs/latest/web-ui.html 93. Monitoring and Instrumentation - https://spark.apache.org/docs/latest/monitoring.html 94. explain() Function - https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.explain.html 95. Schema Evolution - https://spark.apache.org/docs/latest/sql-data-sources-parquet.html#schema-merging 96. Dynamic Partition Overwrite - https://spark.apache.org/docs/latest/sql-data-sources-parquet.html#partition-discovery 97. Partition Overwrite Mode - https://spark.apache.org/docs/latest/sql-data-sources-load-save-functions.html#partition-discovery 98. Memory Management - https://towardsdatascience.com/basics-of-apache-spark-configuration-settings-ca4faff40d45 99. Kryo Serialization - https://spark.apache.org/docs/latest/tuning.html#data-serialization 100. Serialization - https://spark.apache.org/docs/latest/configuration.html#compression-and-serialization 101. SQL Temp Views - https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.createOrReplaceTempView.html 102. Broadcast Timeout - https://spark.apache.org/docs/latest/sql-performance-tuning.html#broadcast-timeout ### Additional Resources 103. PySpark Tutorial - https://sparkbyexamples.com/pyspark-tutorial/ 104. Databricks PySpark Guide - https://docs.databricks.com/aws/en/pyspark/ 105. Apache Spark Performance - https://www.chaosgenius.io/blog/spark-performance-tuning/ 106. PySpark Best Practices - https://towardsdatascience.com/spark-essentials-how-to-read-and-write-data-with-pyspark-5c45e29227cd 107. PySpark SQL Complete Guide - https://codecut.ai/pyspark-sql-complete-guide/

On This Page

On This Page