Databricks and PySpark interview questions

An architecture that combines data lake storage with warehouse features like ACID transactions and schema enforcement, using open formats like Delta.

The driver plans the job and coordinates tasks. Executors run the tasks on partitions of data.

Transformations build a plan and run only when an action like count, write or collect is called.

Transformations (select, filter, join) return a new DataFrame lazily. Actions (count, show, write) trigger execution.

Narrow ones work within a partition (filter, select). Wide ones need data from other partitions and cause a shuffle (groupBy, join).

Moving data between executors over the network and disk. It is often the slowest part of a job.

Sending a small table to every executor so the big table does not shuffle.

from pyspark.sql.functions import broadcast
df = big.join(broadcast(small), 'id')

repartition shuffles to any number of partitions. coalesce only reduces partitions without a full shuffle.

One key has far more rows, so one task runs much longer. Fix with AQE skew handling, filtering NULL keys, or salting.

Spark changes the plan at runtime: merges small partitions, switches join types and splits skewed partitions.

A storage layer on Parquet that adds ACID transactions, time travel, MERGE and schema enforcement.

With MERGE.

MERGE INTO silver.orders t USING updates s
ON t.order_id = s.order_id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;

OPTIMIZE compacts small files. VACUUM removes old files no longer needed (after the retention period).

Bronze (raw), silver (cleaned and conformed) and gold (business-ready) layers.

Incremental file ingestion that detects new files in cloud storage and processes each file once.

Central governance for Databricks: catalogs, schemas, permissions, lineage and external locations across workspaces.

Each row moves between the JVM and Python, which is slow. Built-in functions run inside Spark's engine.

cache keeps a DataFrame in memory (default level). persist lets you choose the storage level. Only cache data you reuse, and unpersist it.

Job clusters start for a job and stop after, which is cheaper. All-purpose clusters are for interactive work.

Open the Spark UI, find the longest stage, check spill and the gap between median and max task time, then fix skew, joins or file sizes.