Data Engineer Top 50 Q&A

A Data Warehouse stores structured data processed for a specific purpose. A Data Lake stores raw, unstructured, or semi-structured data at scale.

ETL (Extract, Transform, Load) transforms data before loading it into the warehouse. ELT (Extract, Load, Transform) loads raw data first and leverages the warehouse's power to transform it.

Star Schema has a central fact table connected to denormalized dimension tables. Snowflake Schema normalizes dimension tables, saving space but increasing join complexity.

A method to track historical data by creating new records for changes, using start/end dates or versions to identify the current record.

Conceptual: High-level business view. Logical: Detailed structure without platform specifics. Physical: Actual implementation on a specific DB platform.

A unique identifier produced by the system (usually an integer) that has no business meaning, used as a primary key in dimension tables.

OLTP (Online Transactional Processing) is for fast, short transactions (like banking). OLAP (Online Analytical Processing) is for complex queries and data analysis.

Horizontal partitioning of data across multiple databases to improve scalability and performance.

A decentralized architectural pattern where data is managed by domain-specific teams as a product.

A set of software design patterns used to determine and track the data that has changed so that action can be taken using the changed data.

A unified analytics engine for large-scale data processing. It's much faster than MapReduce because it processes data in-memory.

RDD is the basic building block (unstructured). DataFrame is a distributed collection of data organized into named columns (structured). Dataset is a type-safe interface to DataFrames.

Transformations (like map, filter) are lazy; they build a logical plan. Actions (like count, collect) trigger the actual computation.

Spark doesn't run transformations immediately. It records them and optimizes the execution plan when an action is called.

Directed Acyclic Graph. The sequence of computations performed on data in Spark, which can be optimized before execution.

Data Skew is when data is unevenly distributed across partitions. Fixes: Salting the keys, repartitioning, or using broadcast joins.

When joining a large table with a small table, Spark sends the small table to all executors to avoid expensive data shuffles.

An extension of the Spark API that allows scalable, high-throughput, fault-tolerant stream processing of live data streams.

The Driver coordinates the job and manages the DAG. Executors perform the actual data processing tasks.

Both are used to store intermediate results in memory. Cache uses the default storage level (MEMORY_ONLY), while Persist allows you to specify other levels (e.g., DISK_ONLY).

Row-based (like CSV, Avro) is good for transactional writes. Columnar (like Parquet, ORC) is much faster for analytical queries that read specific columns.

An open-source, columnar storage file format for Hadoop. It provides efficient data compression and encoding schemes.

A row-based data serialization system that uses JSON for defining data types and protocols, and serializes data into a compact binary format.

An open-source storage layer that brings ACID transactions to Apache Spark and big data workloads.

Hadoop Distributed File System. A distributed file system designed to run on commodity hardware and provide high-throughput access to data.

A central repository of Apache Hive metadata, storing information about tables, partitions, and their storage locations.

Non-relational databases. Types: Document (MongoDB), Key-Value (Redis), Column-family (Cassandra), Graph (Neo4j).

A cloud-based data platform with a unique multi-cluster shared data architecture that separates storage, compute, and services.

Partitioning splits the table into physical chunks based on a column. Indexing creates a separate data structure to speed up data lookup.

Amazon Simple Storage Service. Often used as the storage layer for Data Lakes due to its durability and scalability.

A platform to programmatically author, schedule, and monitor workflows using Directed Acyclic Graphs (DAGs).

A collection of all the tasks you want to run, organized in a way that reflects their relationships and dependencies.

Templates for a single task in a workflow (e.g., PythonOperator, BashOperator, S3ToRedshiftOperator).

A distributed event store and stream-processing platform capable of handling high-throughput, real-time data feeds.

Topic: A category name where messages are stored. Producer: Publishes messages to topics. Consumer: Reads messages from topics.

The property that running a pipeline multiple times with the same input yields the same result, ensuring reliability during retries.

Using retries, alerts, dead-letter queues for bad data, and ensuring tasks are idempotent.

Automated checks to ensure data completeness, accuracy, consistency, and timeliness (e.g., using Great Expectations).

Docker containers package dependencies for portability. Kubernetes orchestrates and scales these containers for large-scale data workloads.

A tool that allows data analysts and engineers to transform data in their warehouse using SQL and software engineering best practices (like version control and testing).

Source (DB/App) -> Kafka -> Spark Streaming/Flink -> Warehouse (Snowflake/BigQuery) -> Dashboard.

Check for data skew, increase executor memory, optimize joins (e.g., use broadcast), or reduce the number of partitions.

Using file formats like Parquet/Avro that support schema merging, or using a schema registry in Kafka.

Discuss encryption at rest/transit, data masking, access control (RBAC), and ensuring PII is handled according to policy.

Check the query profile, use clustering keys, optimize the use of result caching, and ensure the warehouse size is appropriate.

Focus on the scale, the tools used (Airflow, Spark, Kafka), and the business value it provided.

By using MVP approaches for speed while building automated data quality checks to ensure long-term quality.

Due to its ease of use, massive data library ecosystem (Pandas, PySpark, Airflow), and faster development cycles.

Using tools like dbt docs, data dictionaries, and maintaining clear DAG code comments.

Moving towards Data Contracts, Data Observability, and automated AI-driven pipeline optimization.