Microsoft Azure Data Engineer Interview Questions

Microsoft Fabric is a unified SaaS analytics platform with OneLake as its single storage layer for all workloads (Lakehouse, Warehouse, Real-time Analytics, Power BI). OneLake is tenant-wide and automatically shared across all Fabric workspaces with zero data copying. ADLS Gen2 requires manual mounting and explicit data movement between services. OneLake uses shortcuts to link external ADLS, S3, or GCS storage.

Azure Databricks: Unity Catalog for governance, MLflow for ML experiment tracking, Delta Live Tables for declarative pipelines, advanced autoscaling with spot instance optimization. Synapse Spark Pools: native Synapse Pipelines integration, shared metadata with Synapse SQL, no additional licensing cost. Choose Databricks for ML engineering and complex streaming. Choose Synapse for SQL-first teams already in Synapse. Microsoft strategic direction now favors Fabric for net-new projects.

Three patterns: 1) Watermark — store last processed timestamp in control table, query WHERE updated_at > last_watermark, update watermark post-run. 2) Change Data Capture (CDC) — native CDC on Azure SQL/Cosmos DB using LSN tracking, no watermarks needed. 3) Partition-based — process only new date partitions with ForEach activity. Combine CDC for operational sources with watermark for API sources that do not expose change logs.

The _delta_log folder contains JSON commit files recording every operation: AddFile (new Parquet files added), RemoveFile (files logically deleted), Metadata (schema changes). Atomicity: a transaction either fully writes its commit JSON or does not appear — readers never see partial writes. Isolation: optimistic concurrency control checks conflicts at write time. Consistency: schema validation enforced at commit. Durability: commit JSON in blob storage is durable. Time travel uses commit history to reconstruct any past version.

Step 1: ADF Monitor tab — review failed activity run error message and input/output JSON. Step 2: Enable diagnostic logging to Log Analytics Workspace, query PipelineRuns and ActivityRuns with KQL for error trends. Step 3: Configure Retry policy — 3 retries with 30s delay for transient network errors, disabled for schema validation errors. Step 4: Set Azure Monitor Alert Rules sending notifications to Teams or PagerDuty. Step 5: For Data Flows, enable debug mode and use Inspect pane to preview column values at each transformation step.

Z-Order co-locates related rows in the same Parquet files by combining multiple column values into a space-filling curve. Run: OPTIMIZE delta./path ZORDER BY (country, product_id). Use when queries filter on multiple columns together (WHERE country = 'India' AND product_id = 'P123'). Do not Z-Order single columns — use regular partitioning instead. Do not Z-Order high-cardinality columns like GUIDs. Microsoft recommends Z-Order for columns appearing in more than 50% of WHERE clauses.