01What is Azure Data Factory?
A cloud service to orchestrate and move data with pipelines, activities, linked services and datasets.
Data engineer interview questions for freshers
Pipelines, data lakes, Spark and Azure questions for your first data engineering job, with answers a junior is expected to give. 30 questions with answers, written by a Senior Data Engineer.
How to prepare in one week
- Know the basics cold. Freshers are judged on fundamentals: joins, grouping, data types, and why you would choose one approach over another.
- Practise out loud. Say each answer in under a minute, then give one small example. Interviewers remember examples, not definitions.
- Write code by hand. Solve a few problems without autocomplete. Our free practice editors check your answer instantly.
- Show one project. Prepare a 2 minute story about one project: the data, what you built, one problem you solved and the result.
- Do a mock interview. Our AI mock interview asks 8 questions for your role and scores each answer. The first one is free.
- Fix your resume and LinkedIn. Run the free ATS resume checker and LinkedIn optimizer against the job post before you apply.
The questions, with answers
02Linked service vs dataset?
A linked service is the connection (like a connection string). A dataset points to specific data on that connection, like a table or folder.
03What is an integration runtime?
The compute ADF uses: Azure IR for cloud data, self-hosted IR for on-premises or private networks.
04How do you build a metadata-driven pipeline?
Keep a control table of sources and targets, read it with Lookup, loop with ForEach and pass parameters to a generic Copy activity.
05How do you do incremental loads?
Store a watermark (last modified date or ID), load only newer rows, then update the watermark.
06Types of triggers in ADF?
Schedule, tumbling window (with dependencies and backfill), storage event and custom event triggers.
07What is ADLS Gen2?
Azure Blob Storage with a hierarchical namespace, built for analytics with folder-level ACLs.
08RBAC vs ACLs in ADLS?
RBAC grants access at account or container level. ACLs give fine-grained access on folders and files.
09How should a data lake be organised?
Zones like raw/bronze, curated/silver and serving/gold, partitioned by date where it helps, using Parquet or Delta.
10What is a managed identity?
An identity Azure manages for a service, so it can access other resources without storing passwords.
11What is the lakehouse?
An architecture that combines data lake storage with warehouse features like ACID transactions and schema enforcement, using open formats like Delta.
12Driver vs executors?
The driver plans the job and coordinates tasks. Executors run the tasks on partitions of data.
13What is lazy evaluation?
Transformations build a plan and run only when an action like count, write or collect is called.
14Transformation vs action?
Transformations (select, filter, join) return a new DataFrame lazily. Actions (count, show, write) trigger execution.
15Narrow vs wide transformations?
Narrow ones work within a partition (filter, select). Wide ones need data from other partitions and cause a shuffle (groupBy, join).
16What is a shuffle and why is it expensive?
Moving data between executors over the network and disk. It is often the slowest part of a job.
17What is a broadcast join?
Sending a small table to every executor so the big table does not shuffle.
from pyspark.sql.functions import broadcast
df = big.join(broadcast(small), 'id')18repartition vs coalesce?
repartition shuffles to any number of partitions. coalesce only reduces partitions without a full shuffle.
19What is data skew and how do you fix it?
One key has far more rows, so one task runs much longer. Fix with AQE skew handling, filtering NULL keys, or salting.
20What is Adaptive Query Execution?
Spark changes the plan at runtime: merges small partitions, switches join types and splits skewed partitions.
21Structured vs semi-structured vs unstructured data?
Structured fits tables, semi-structured has flexible fields like JSON, unstructured has no fixed format like images and documents.
22OLTP vs OLAP?
OLTP handles many small transactions. OLAP handles large analytical queries over history.
23Batch vs streaming?
Batch processes data in groups on a schedule. Streaming processes events continuously as they arrive.
24What is normalisation?
Splitting data into related tables to reduce duplication and keep it consistent.
25DDL vs DML?
DDL defines structure (CREATE, ALTER, DROP). DML changes data (INSERT, UPDATE, DELETE).
26Azure SQL Database vs Managed Instance vs SQL on a VM?
SQL Database is fully managed PaaS for one database. Managed Instance is near full SQL Server compatibility as PaaS. SQL on a VM is IaaS with full control.
27What open-source databases does Azure offer?
Azure Database for PostgreSQL and Azure Database for MySQL as managed services.
28What is Azure Cosmos DB?
A globally distributed NoSQL database with low latency and several APIs, including NoSQL, MongoDB, Cassandra, Gremlin and Table.
29What is a partition key in Cosmos DB?
The property used to spread data across partitions. A good one has many distinct values and spreads load evenly.
30Cosmos DB consistency levels?
Strong, bounded staleness, session, consistent prefix and eventual, trading consistency for latency and availability.
Keep going
Ready for the real interview?
Practise with our AI interviewer: 8 questions for your role, a score for every answer and a stronger answer to learn from.