Most ML tutorials jump straight into models and algorithms. But here's the thing no one tells beginners: the model is not where the magic happens. The data preparation is.

Think of it this way. You wouldn't hand a toddler a jigsaw puzzle with missing pieces, warped edges, and pieces from three different puzzles mixed, and then blame the toddler for not solving it. That's exactly what happens when we feed raw, messy data into a machine learning model and wonder why it performs terribly.

Data preprocessing is the act of preparing raw data so a model can actually make sense of it. It covers everything from creating useful features to handling gaps in the data to making sure numbers play fair with each other. In this piece, we’ll walk through four pillars of preprocessing — Feature Engineering, Handling Missing Values, Scaling, and Normalization. Not as dry mathematical procedures, but as intuitive steps you’ve already seen in how children learn to organize their world.

By the end, you won’t just know what these terms mean. You’ll understand why they exist, when they matter, and what breaks when you skip them.

Why Preprocessing Matters

A machine learning model is only as good as the data it receives.

Raw data from the real world is messy. It has gaps. It has inconsistencies. Some numbers are in the thousands, others are tiny decimals. Some information is buried inside other information and needs to be extracted before it becomes useful.

Preprocessing is the bridge between raw chaos and structured learning.

The Four Pillars of Data Preprocessing

Each pillar solves a specific problem in your data. Skip any one of them, and your model pays the price.

1. Feature Engineering:

Teaching the Machine What to Notice

• The Child: A child walks into a pet store. They see animals everywhere. But their parents point and say, “Look at the size. Look at the fur. Does it have a tail?” The parent isn’t changing the animals; they’re teaching the child what to pay attention to.

• The Machine: Raw data often contains information, but not in a form the model can use directly. Feature engineering is the process of creating, transforming, or selecting the right input variables (features) so the model can actually find patterns.

• Why it matters: A column called “date_of_birth” is useless to a model predicting insurance risk. But a new column called “age” was derived from that date? Now the model has something it can work with.

What Feature Engineering Looks Like in Practice

Creating new features from existing ones: You have a “timestamp” column. You extract “hour_of_day”, “day_of_week”, and “is_weekend”. Suddenly, a model predicting restaurant traffic has useful signals instead of a raw timestamp it can’t interpret.

Combining features: You have “house_length” and “house_width”. Neither alone tells the full story. Multiply them to get “house_area”, a single feature that captures what two couldn’t.

Encoding categories: A column says “Red”, “Blue”, “Green”. A model doesn’t speak English. You convert these to numbers the model can process; one-hot encoding turns “Red” into [1, 0, 0], “Blue” into [0, 1, 0], and so on.

The Takeaway: Feature engineering is the art of translating human knowledge into a language the model can learn from. The model finds patterns, but you decide what it gets to look at.

2. Handling Missing Values:

Filling in the Gaps

• The Child: A child is reading a picture book, and one page is torn out. They don’t throw away the entire book. They might guess what happened on that page based on the story before and after. Or they skip it and keep reading.

• The Machine: Real-world datasets almost always have missing values. Sensors fail. People skip survey questions. Records get corrupted. The model needs a strategy: fill the gap, remove the gap, or flag the gap.

• Why it matters: Most ML algorithms cannot process empty cells. If you don’t handle missing values, your model either crashes or silently learns the wrong thing.

The Three Strategies for Missing Data

Strategy 1 — Remove it (Drop): If only a tiny fraction of your data has gaps, sometimes the simplest move is to drop those rows. Like a child skipping the torn page, you lose a little, but the rest of the story still makes sense. Risk: If too many rows have gaps, you lose valuable data.

Strategy 2 — Fill it (Impute): Replace the missing value with something reasonable. The most common approaches: use the mean (average) for numerical data, the mode (most frequent value) for categorical data, or the median if your data has extreme outliers. Like the child guessing what happened on the missing page based on context.

Strategy 3 — Flag it (Indicator): Create a new column that says “this value was missing” (1 or 0). This way, the model knows the gap existed and can learn whether the absence of data is itself a signal. Sometimes, the fact that a patient skipped a question on a health survey is more informative than any answer they could have given.

The Takeaway: Missing data isn’t a disaster; it’s a decision point. How you handle it depends on how much is missing, why it’s missing, and what your model needs.

3. Feature Scaling:

Making the Numbers Play Fair

• The Child: Imagine two children comparing their collections. One child has 3 seashells. The other has 3,000 stickers. If you ask, “Who has the bigger collection?” the answer seems obvious, but is it fair? The scales are completely different. To compare meaningfully, you’d need to put both collections on the same measuring system.

• The Machine: ML models that calculate distances or gradients (like KNN, SVM, or neural networks) are heavily influenced by the magnitude of numbers. A feature ranging from 0 to 1,000,000 will dominate a feature ranging from 0 to 1, even if the smaller feature is more important.

• Why it matters: Without scaling, large-magnitude features bully small-magnitude features into irrelevance. The model doesn’t know that “salary in rupees” and “years of experience” should carry equal weight; it just sees big numbers and small numbers.

The Two Main Scaling Methods

Min-Max Scaling (Normalization to a range): Squishes all values into a fixed range, typically 0 to 1. The formula: (value - min) / (max - min). Like converting both children’s collections to a percentage of their personal maximum. Works well when you need bounded values and your data doesn’t have extreme outliers.

Standardization (Z-score Scaling): Centers the data around 0 with a standard deviation of 1. The formula: (value - mean) / standard_deviation. Like grading on a curve — every student’s score is expressed as “how far from average.” Works well when your data has outliers or when your algorithm assumes normally distributed data.

The Takeaway: Scaling doesn’t change what your data says. It changes how loudly each feature speaks, so no single feature drowns out the others.

4. Normalization:

Reshaping the Distribution

• The Child: A teacher asks the class to rate how much they liked a movie on a scale of 1 to 10. One child always gives everything a 9 or 10. Another child’s ratings spread evenly from 1 to 10. To compare their opinions fairly, you’d need to adjust for each child’s personal rating style and their distribution.

• The Machine: Normalization transforms the shape of your data’s distribution. Even after scaling, your data might be heavily skewed, with most values clustered on one side with a long tail. Many ML algorithms perform better when the data follows a more symmetric, bell-curve-like distribution.

• Why it matters: Skewed distributions can mislead models. If 95% of your income data clusters between ₹20,000 and ₹80,000 but a few values shoot up to ₹50,00,000, the model might overfit to those extremes or underperform on the majority.

Common Normalization Techniques

Log Transformation: Take the logarithm of each value. This compresses the long tail and spreads out the clustered values. Extremely effective for income data, population data, or anything with exponential growth patterns.

Box-Cox Transformation: A more flexible version that finds the optimal power transformation to make your data as close to a normal distribution as possible. The math picks the best “reshaping” automatically.

Quantile Transformation: Forces the data into a specific distribution (usually uniform or normal) by mapping values to their percentile ranks. The most aggressive approach guarantees the output shape but can distort relationships between close values.

The Takeaway: Normalization is about the shape of your data, not the scale. It ensures your data’s distribution doesn’t secretly sabotage your model’s assumptions.

Scaling vs. Normalization:

Clearing Up the Confusion

These two terms get mixed up constantly,

Scaling adjusts the range or magnitude of your features. It answers: “How big are these numbers relative to each other?”

Normalization adjusts the distribution shape of your features. It answers: “What does the spread of these numbers look like?”

You often need both. Scale first to get features on comparable ranges, then normalize if the distribution is skewed.

Think of it as: scaling sets the volume, normalization tunes the equalizer.

The Preprocessing Decision Tree

How do you decide what preprocessing to apply?

Conclusion

Data preprocessing isn’t glamorous. Nobody writes headlines about it. But it is, without exaggeration, where most real ML work happens.

Feature engineering decides what the model gets to see. Missing value handling decides how gaps are managed. Scaling ensures no feature unfairly dominates. Normalization reshapes distributions so algorithms can work as designed.

These aren’t optional steps you can skip when you’re in a hurry. They’re the foundation. Every model, every prediction, every result sits on top of how well you prepared the data.

If you step back, the pattern is strikingly familiar: whether it’s a child learning to organize their toy box or a model learning to classify images, the quality of learning depends entirely on the quality of preparation.

Get the data right, and the model will surprise you. Skip the preparation, and no algorithm in the world will save you.

Most machine learning explanations start with algorithms. In this blog lets start with a simpler question: how does anything learn?

At its core, machine learning isn’t magic; it’s a set of strategies for turning data into behavior. Sometimes we guide the system with explicit answers. Sometimes we let it uncover structure on its own. And sometimes we let it learn through feedback, improving with every decision it makes.

In this piece, we’ll walk through the four core learning paradigms: supervised, unsupervised, self-supervised, and reinforcement learning, not as abstract theory, but as intuitive learning patterns you’ve already seen in the real world. By the end, you won’t just recognize the terms, you’ll understand why these approaches exist, when they matter, and how they shape modern AI systems.

Defining the Paradigms

The learning paradigms are the fundamental way an algorithm processes data to find patterns. It is not just a mathematical function; it is a distinct philosophical approach to problem-solving.

The Four paths to pattern recognization

Each paradigm dictates exactly what kind of data the machine needs and how independently it is allowed to operate.

1. Supervised Learning: The Flashcard Method

• The Child: You show a child a card with a picture of a 🍎 and the word “Apple” written on it. You repeat this until they can see the shape and say the word.

• The Machine: We give the model a Dataset (the pictures) and Labels (the word “Apple”). The model calculates a mathematical function to map the input to the correct output.

• Use case: This is used for Classification (Is this email spam or not?) and Regression (Predicting the exact price of a house).

Ground Truth: The model relies entirely on the “Teacher” being 100% correct on the labels provided.

2. Unsupervised Learning: Discovering on Your Own

• The Child: A child is given a bucket of colored blocks 🟦🧱🟡. Without being told to, they might start putting all the red ones together and all the blue ones together.

• The Machine: The model looks for Features (color, shape, size) and calculates the “distance” between data points. Things that are “close” together get grouped into Clusters.

• Use case: This is used for Clustering (Grouping similar news articles together) and Anomaly Detection (Finding a credit card transaction that looks "weird" or "different" from your usual spending).

Clustering: Finding underlying structure in chaotic data when you don’t know what you are actually looking for yet.

3. Self-Supervised Learning: The Context Clue

• The Child: You show a child a sentence with a word missing: “The cat sat on the ___.” Based on their past experience, they guess “mat” or “floor.”

• The Machine: We take a massive amount of unlabeled data (like the entire internet) and hide parts of it. The model tries to predict the missing piece.

• Use case: This is the "secret sauce" behind Large Language Models (LLMs) like ChatGPT. It allows machines to learn from massive amounts of data without humans having to label every single thing.

Pre-training: This is how "Brainy" models like GPT are built before they are taught specific tasks.

4. Reinforcement Learning: Learning from Consequences

• The Child: A child tries to ride a bike 🚲. They wobble (penalty), they adjust their balance, and eventually, they move forward (reward).

• The Machine: An Agent lives in an Environment. It takes an Action, and the environment gives it a Score (Reward). The machine’s only goal is to maximize that score over time.

• Use case: This is used for Robotics and Game AI (like AlphaGo), where the machine needs to make a sequence of decisions to reach a goal.

Trial and Error: The model doesn't need to be told the "right" answer upfront, just whether it’s action are moving it closer to the ultimate goal.

The Architect’s Decision Tree

How to choose the right paradigm for your specific problem?

Conclusion

Machine learning, at its core, is not about models; it’s about how learning is structured.

Supervised learning shows us the power of guidance. Unsupervised learning reveals how structure can emerge without labels. Self-supervised learning bridges the gap, turning raw data into its own teacher. And reinforcement learning pushes systems to learn through interaction, feedback, and long-term consequences.

These paradigms aren’t competing ideas; they are complementary lenses. Modern AI systems often blend them, moving fluidly from labeled data to pattern discovery to autonomous decision-making. Understanding this progression is what separates surface-level familiarity from true intuition.

If you step back, the pattern is strikingly simple: whether in machines or humans, learning evolves from instruction → exploration → self-discovery → adaptation.

And once you see machine learning this way, every model, every system, and every breakthrough becomes easier to reason about, not as complexity, but as a variation of these fundamental ways of learning.

In today’s data-driven world, the ability to process massive amounts of information in near real-time is a significant competitive advantage. For businesses in sectors such as finance, e-commerce, and IoT, handling millions of transactions per second is no longer a futuristic dream; it’s a daily necessity. But how do you build a system that can handle this data deluge without buckling under pressure?

This guide will walk you through the fundamentals of designing a high-throughput, low-latency data pipeline. We’ll explore the initial questions to ask, the core architectural decisions, and a detailed look at the modern technologies that power today’s most demanding data platforms.

First Things First: Asking the Right Questions

Before diving into the technical nitty-gritty, it’s crucial to understand the “what” and “why” of your data pipeline. Rushing this stage is like building a house without a blueprint. Here are some essential questions to evaluate your needs:

• What is the nature of your data? Is it structured (like financial transactions), semi-structured (like JSON logs), or unstructured (like social media feeds)?

• What are your throughput and latency requirements? Be specific. Do you need to process 1 million transactions per second with a 100-millisecond delay, or 10 million with a sub-50-millisecond delay?

• Is real-time processing a must-have? Can some data be processed in batches, or must every event be acted upon instantly?

• What level of data accuracy and reliability is acceptable? Do you need an “exactly-once” processing guarantee to ensure every single transaction is accounted for?

• What are your scalability and future growth projections? Your pipeline should be able to handle not just today’s data load, but also your expected growth.

• What is your budget and in-house expertise? Are you looking for open-source solutions that require more hands-on management, or are you willing to invest in managed cloud services?

The Core Architectural Decisions

Once you have clear requirements, you can start making fundamental design choices.

Batch vs. Stream Processing: The Eternal Debate

• Batch Processing: Data is collected over a period and then processed in large chunks or “batches.” This is efficient for large volumes of data where immediate action isn’t necessary, like end-of-day financial reporting.

• Stream Processing: This method involves processing data in real-time as it’s generated. For use cases like real-time fraud detection or instant e-commerce recommendations, stream processing is non-negotiable.

For many modern applications that require both real-time insights and deep historical analysis, a hybrid approach is often employed. Architectures like the Lambda Architecture combine both batch and stream processing layers to serve different needs. A simpler alternative, the Kappa Architecture, relies solely on stream processing.

Choosing Your Technology Stack: The Building Blocks

A data pipeline is composed of several interconnected stages. Here’s a breakdown of the key components and popular technology choices for each:

1. Data Ingestion: The Front Door

This is where your data enters the pipeline. The goal here is to reliably capture a high volume of data from various sources.

• Key Technologies:

  • Apache Kafka: A distributed streaming platform that has become the standard for high-throughput, real-time data feeds. It acts as a durable and scalable “message bus.”

  • Amazon Kinesis: A fully managed service on AWS for real-time data streaming.

  • Google Cloud Pub/Sub: A serverless messaging service on Google Cloud that can handle millions of messages per second.

2. Data Processing: The Brains of the Operation

This is where the magic happens. The processing engine transforms, enriches, and analyzes the incoming data streams.

• Key Technologies:

  • Apache Flink: A powerful open-source framework for stateful stream processing, known for its high performance and low latency.

  • Apache Spark Streaming: An extension of the popular Apache Spark analytics engine that enables scalable and fault-tolerant stream processing.

  • Google Cloud Dataflow: A managed service that provides a unified model for both batch and stream processing.

3. Data Storage: The Modern Data Platform

Once processed, the data needs to be stored for querying, analysis, and visualization. The choice of storage depends on how you intend to use the data. This is where modern data architectures like the Data Lakehouse come into play, blending the flexibility of a data lake with the power of a data warehouse.

The Rise of the Lakehouse: Hudi, Delta Lake, and Iceberg

Traditional data lakes, while cost-effective for storing vast amounts of raw data on platforms like Amazon S3, often struggle with reliability and performance. This has led to the emergence of the “lakehouse” architecture, which brings data warehouse capabilities like ACID transactions and schema enforcement directly to your data lake. This is powered by open table formats that manage the underlying data files.

Apache Hudi

• What it is: Standing for “Hadoop Upserts, Deletes and Incrementals,” Hudi is an open-source transactional data lake platform that specializes in enabling low-latency database-style operations on the data lake.

• How it helps: Hudi is excellent for change data capture (CDC) scenarios, allowing for efficient upserts (updating existing records) and deletes. It provides a choice between optimizing for write performance (Copy-on-Write) or query performance (Merge-on-Read), giving engineers fine-grained control over their pipeline’s tradeoffs.

• Key Features:

  • Incremental Processing: Provides powerful primitives for processing only the data that has changed since the last run.

  • Copy-on-Write vs. Merge-on-Read: Two distinct storage types to optimize for specific read/write patterns.

  • Concurrency Control: Manages simultaneous writers and readers to ensure data consistency.

Delta Lake

• What it is: An open-source storage layer that brings ACID transactions to Apache Spark and other big data engines. It adds a transaction log on top of your Parquet files in your data lake.

• How it helps: In a high-throughput pipeline, data is constantly being written. Delta Lake ensures that your queries always see a consistent view of the data, even during writes. It helps manage the “small file problem” by optimizing file sizes, which is crucial for query performance. Its deep integration with Spark Structured Streaming makes it a natural fit for real-time pipelines.

• Key Features:

  • ACID Transactions: Guarantees data reliability and consistency for concurrent operations.

  • Time Travel: Allows you to query previous versions of your data, invaluable for debugging and auditing.

  • Schema Enforcement & Evolution: Prevents bad data from corrupting tables and allows schemas to change over time.

Apache Iceberg

• What it is: An open table format for huge analytic datasets, originally developed at Netflix. Like its counterparts, it brings the reliability of SQL tables to your data lake.

• How it helps: Iceberg is known for its strong schema evolution capabilities, allowing you to add, drop, or rename columns without rewriting data files. This is a massive advantage in fast-moving environments where data structures change. Its design avoids performance bottlenecks tied to a central metadata store, making it highly scalable.

• Key Features:

  • Full Schema Evolution: Adapt your table schemas to evolving data sources without disrupting the pipeline.

  • Hidden Partitioning: Simplifies querying by automatically handling data partitioning.

  • Engine Agnostic: Designed to work with various processing engines like Spark, Flink, and Presto, preventing vendor lock-in.

The Powerhouses: Managed Cloud Platforms

For organizations that prefer a fully managed experience, cloud-native platforms offer incredible power and ease of use.

Snowflake

• What it is: A fully managed, cloud-native data platform that can act as a data warehouse, data lake, or a combination of both.

• How it helps: Snowflake’s unique architecture separates storage from compute, meaning you can scale your processing power up or down independently to handle fluctuating workloads without disruption. For high-throughput ingestion, it offers tools like Snowpipe Streaming for low-latency, real-time data loading.

• Key Features:

  • Separation of Storage and Compute: Provides incredible flexibility and cost-efficiency.

  • High Concurrency: Handles a large number of concurrent users and queries without performance degradation.

  • Semi-Structured Data Support: Natively handles formats like JSON and Avro.

Other major players include Google BigQuery and Amazon Redshift, which are powerful data warehouses optimized for complex analytical queries on large datasets, and Databricks, a unified platform built around Apache Spark and Delta Lake that aims to cover the entire data and AI lifecycle.

For Specialized Access Patterns

• For Low-Latency Access: In-memory databases like Redis or Apache Ignite offer the fastest possible access times for applications that need immediate responses, such as caching or real-time feature stores. NoSQL databases like Apache Cassandra or DynamoDB are designed for high-throughput reads and writes at scale.

• For High-Performance Needs: For the most demanding AI and real-time analytics workloads, leveraging NVMe (Non-Volatile Memory Express) Storage can offer extremely high-speed I/O.

Putting It All Together: A Sample Architecture

Imagine an e-commerce platform that needs to provide real-time product recommendations and detect fraudulent transactions. A modern pipeline could look like this:

1. Ingestion: User clickstream and transaction events are sent to Apache Kafka.

2. Processing: An Apache Flink job consumes the Kafka streams in real-time, enriching and analyzing the data.

3. Storage & Serving:

   • The processed data is continuously written to an Apache Hudi table on Amazon S3 to efficiently handle frequent updates to user profiles and order statuses.

   • A fraud detection model flags suspicious transactions and sends alerts immediately.

   • For broad business intelligence, the Hudi tables are queried using Snowflake, which can elastically scale its compute to meet query demands from different teams without impacting the real-time pipeline.

   • The latest product recommendations are pushed to a low-latency Redis cache for instant retrieval by the website.

The Journey Doesn’t End Here

Building a high-throughput, low-latency data pipeline is a complex but achievable goal. By starting with a clear understanding of your requirements and leveraging modern open table formats like Hudi, Delta, and Iceberg, alongside powerful cloud platforms like Snowflake, you can create a robust, scalable, and future-proof system. Remember that a data pipeline is not a one-and-done project; it requires continuous monitoring, optimization, and iteration to meet the evolving needs of your business.

A file format defines how data is structured and stored so that software can read and write it consistently.

Think of it as the container for your data. Just like music comes in MP3, WAV, or FLAC, data files also come in different formats depending on the use case.

Common data file formats today:

• CSV – Simple text-based, rows separated by commas (easy to use, but large in size and no compression).

• JSON – Flexible, hierarchical (good for APIs but verbose).

• Avro – Row-based binary format, compact, supports schema evolution (common in streaming).

• ORC – Columnar format optimized for Hadoop and Hive.

• Parquet – Columnar format designed for analytics and compression efficiency.

Introduction to Parquet

If you work with data, you’ve probably heard the word Parquet thrown around in conversations about data lakes, analytics, or cloud storage. But what exactly is it, and why does everyone seem to love it?

The Everyday Analogy: Organizing Your Groceries

Imagine you run a grocery delivery service, and you have to store all the items customers ordered.

You could store them in two ways:

1. Row-based storage (like CSV)

   You put one of each item from an order in a bag: milk, bread, apples, pasta — all together. If you want to find all the bread you’ve sold, you have to open every bag and dig around to see if bread is inside.

2. Column-based storage (like Parquet)

   Instead of mixing, you group similar items together:

   • One box for all the milk

   • One box for all the bread

   • One box for all the apples

   If you need all the bread you’ve sold, you simply open the bread box — no need to touch anything else.

That’s Parquet in action, organized storage that makes retrieval faster and more efficient.

The Technical View

Parquet is an open-source, columnar storage format developed by Twitter and Cloudera, now an Apache project. It’s:

• Self-describing – Metadata in the file footer stores schema, number of rows, and statistics.

• Highly compressed – Supports Snappy, Gzip, ZSTD, and more.

• Structured internally into:

  • Row Groups → subsets of rows (horizontal partitions).

  • Column Chunks → data for each column inside a row group (vertical partitions).

  • Pages → the smallest unit of storage, making reads efficient.

• Magic Number (PAR1) – Ensures the file is recognized as Parquet.

This hybrid layout balances efficient columnar reads with manageable write performance.

Why Choose Parquet Over Other Formats?

Now that we understand Parquet, let’s see why it stands out compared to other formats.

• CSV – Human-readable but inefficient; no compression or schema support.

• JSON – Flexible for nested data, but verbose and large in size.

• Avro – Great for row-based streaming but not optimized for analytics.

• ORC – Similar to Parquet, optimized for Hadoop and Hive workloads.

Parquet vs the rest:

• Much smaller file sizes due to efficient compression

• Faster query performance with column pruning and predicate pushdown

• Widely supported across Spark, Hive, Presto, Trino, BigQuery, Snowflake, AWS Athena, and more

• Ideal for data lakes and analytical workloads

How Data is Written in Parquet

When writing data into a Parquet file, the process follows a layered structure:

1. Rows are collected into Row Groups

   • Instead of writing each record one by one (like CSV), Parquet groups multiple rows together into a Row Group (horizontal partition).

2. Within each Row Group, data is split into Column Chunks

   • Each column’s values are stored together → Column A values go into Column Chunk A, Column B values into Column Chunk B, and so on.

   • This preserves the columnar storage advantage.

3. Column Chunks are broken into Pages

   • Pages are the smallest unit of storage in Parquet.

   • Types of pages:

     • Data Pages → actual column data

     • Dictionary Pages → dictionary encoding for repeated values

     • Index Pages (optional) → help with seeking

4. Metadata is written into the File Footer

   • Contains schema, row count, statistics (min/max per column), and row group metadata.

   • Makes Parquet a self-describing format.

5. Magic Number (PAR1) is added

   • At the start and end of the file → ensures the file can be validated as a Parquet file.

How Data is Read from a Parquet File

1. File Footer Read First

• Parquet doesn’t scan the entire file right away.

• It jumps directly to the footer, where all the metadata is stored.

• Metadata tells the engine:

  • Schema (what columns exist)

  • Number of rows

  • Statistics for each column in each row group (min/max)

  • Row group and column chunk offsets (where the data lives in the file)

➡️ This helps the engine decide which parts of the file to actually read.

2. Predicate Pushdown & Column Pruning

• If the query only needs certain columns (SELECT name, age), only those column chunks are accessed.

• If the query has filters (WHERE age > 30), Parquet uses min/max stats in metadata to skip entire row groups that don’t match.

➡️ This reduces I/O drastically compared to row-based formats.

3. Reading Row Groups

• The engine opens only the relevant row groups.

• Inside each row group, it locates the column chunks needed for the query.

4. Decompression & Decoding Pages

• Column chunks are split into pages.

• Each page is decompressed (Snappy, Gzip, ZSTD, etc.) and decoded (Dictionary, RLE, Delta encoding).

• Pages are read into memory one at a time, which is efficient.

5. Reconstructing Results

• Once relevant column data is read, decompressed, and decoded, the engine reconstructs the requested rows and columns.

• The final output is materialized as a result set for the query.

Parquet File Size: Small vs Large

The size of each Parquet file matters a lot for performance.

• Too Small Files (< 50 MB each)

  • Too many metadata and file-open operations.

  • Increases overhead for query engines (Spark, Presto, Hive).

  • Known as the small files problem.

• Too Large Files (> 1 GB each)

  • Slows down reads and reduces parallelism.

💡 Ideal File Size:

• 256 MB to 1 GB for most big data frameworks.

• Large enough to compress well, small enough for parallelism.

How Data Types Affect Compression (and File Size)

Parquet’s efficiency is heavily influenced by data type selection:

• Strings vs Integers: Integers compress far better than storing numbers as strings.

• Decimals vs Floats: Decimals (fixed precision) often compress better.

• Categorical Data: Repeated values (like “USA”) compress extremely well with dictionary encoding.

• Timestamps: Storing as INT64 (epoch values) is more efficient than strings.

Tip: Choose the most precise and smallest data type possible to save space and improve query speed.

Encoding Strategies used in Parquet

1. Run Length Encoding (RLE)

• How it works: Stores consecutive repeated values as a single value + a count.

• Best for: Columns with long runs of the same value (e.g., flags, booleans, country/state codes).

  Example:

  Raw → A A A A B B C C C

  RLE → (A,4), (B,2), (C,3)

• Benefit: Extremely efficient for repetitive data, reduces size drastically.

2. Dictionary Encoding

• How it works: Builds a dictionary of unique values in a column and replaces each value with a small index reference.

• Best for: Low-cardinality columns (few unique values, but repeated often).

  Example:

  Raw → USA, USA, CAN, USA, CAN, MEX

  Dictionary → {0: USA, 1: CAN, 2: MEX}

  Encoded → 0, 0, 1, 0, 1, 2

• Benefit: Saves space when values repeat often; lookups are fast.

3. Delta Encoding

• How it works: Stores the difference (delta) between consecutive values instead of storing full values.

• Best for: Sequential/numeric data like timestamps, IDs, counters.

  Example:

  Raw → 100, 105, 110, 115

  Delta → 100, +5, +5, +5

• Benefit: Great for compressing increasing sequences; works well with integers and dates.

Parquet Optimization & Best Practices

• Predicate Pushdown – Enable in your query engine to filter row groups at scan time.

• Compression Codecs –

  • Snappy: Fast, decent ratio

  • Gzip: Higher compression, slower

  • ZSTD: Good balance

• Avoid Small Files – Merge into larger files to reduce metadata overhead.

• Row Group Size – Balance between compression (bigger row groups) and parallelism (smaller ones).

• Encoding Strategies –

  • Dictionary Encoding for low-cardinality strings

  • Run-Length Encoding for repeated values

  • Delta Encoding for incremental numbers like timestamps

• Sort Data – Sorting on commonly filtered columns improves compression and predicate pushdown.

• ACID Transactions – Parquet itself doesn’t provide ACID; use table formats like Delta Lake, Iceberg, or Hudi for transactional guarantees.

Conclusion

Parquet has become a default choice in the modern data stack because it offers the best of both worlds: compact storage and fast analytics.

By understanding how it works internally (row groups, column chunks, pages) and following best practices (ideal file sizes, data types, compression, and encodings), you can save costs, improve query speeds, and build scalable pipelines.

In short, Parquet is the “organized pantry” of the data world: efficient, structured, and always ready when you need it.

Further Reading

For a deeper dive into the Parquet format’s internals, file layout, and optimizations, check out “The Overview of Parquet File Format” by Vu Trinh. It’s a clear and insightful read for data engineers looking to go beyond the basics.

I worked as a software engineer, mainly focusing on backend development, for a long time, and in 2018, I switched to data engineering, building data pipelines for huge tax datasets at Intuit.

During my initial days, attending meetings felt like walking into a foreign-language class. Terms like data lake, data lineage, and delta lake were thrown around casually, and I was left scribbling them down so I could Google them later.

If you’ve ever found yourself drowning in data lingo, this blog is for you. We’ll break down the most important concepts in today’s data landscape, from the basics like data warehouse to emerging ideas like data mesh, so you can navigate conversations and projects with confidence.

1. Data Collection & Storage

Data Collection

The process of gathering information from various sources to be stored, processed, and analyzed later.

• Sources: Databases, APIs, sensors, user interactions, surveys, web scraping, and third-party datasets.

• Goal: Capture reliable, consistent, and usable data for downstream processes.

• Example: Collecting transaction logs from an e-commerce site, IoT device readings, or customer feedback forms.

Data Ingestion

The process of bringing raw data from multiple sources into a storage system, such as a data lake or data warehouse.

• Types: Batch ingestion (scheduled loads) and streaming ingestion (real-time flows).

• Example: Loading website clickstream data into AWS S3 or Kafka.

Data Lake

A centralized repository that stores raw, unprocessed data — structured, semi-structured, and unstructured at any scale.

• Use Case: Ideal for storing large amounts of data for future analysis.

• Example: AWS S3, Azure Data Lake Storage.

Delta Lake

An open-source storage layer that adds ACID transactions to data lakes, making them reliable for analytics.

• Key Benefit: Handles versioning, schema enforcement, and upserts.

• Example: Databricks Delta Lake.

Data Warehouse

A structured, high-performance database optimized for analytics and reporting. Data here is cleaned, transformed, and stored in a relational format.

• Use Case: Business intelligence dashboards, KPI reporting.

• Example: Snowflake, Google BigQuery, Amazon Redshift.

Data Lakehouse

A hybrid architecture combining the flexibility of a data lake with the reliability and performance of a data warehouse.

• Example: Databricks Lakehouse Platform.

2. Access & Discovery

Data Access Points

The interfaces, services, or endpoints through which users, applications, or systems retrieve or interact with data.

• Examples: APIs, SQL query endpoints, data catalog portals, streaming feeds.

• Importance: Determines how securely and efficiently data consumers can get the information they need.

Data Discovery

The process of finding and cataloging available datasets within an organization.

• Tool Examples: Alation, Collibra, Apache Atlas.

Data Catalog

An organized inventory of datasets, including metadata, lineage, and profiling details.

• Purpose: Makes data assets easier to find and understand.

Metadata

Data about data — describing its source, format, owner, creation date, and usage patterns.

• Example: A dataset’s metadata might include its schema, last refresh date, and responsible data owner.

Data Dictionary

A centralized repository that contains detailed descriptions of all data elements within a database, data warehouse, or system.

• What It Includes: Field names, data types, allowed values, constraints, relationships, and business definitions.

• Purpose: Ensures consistent understanding and usage of data across teams, supports documentation, and helps with governance.

• Example: In a customer database, the data dictionary might specify that Customer_ID is a unique integer, Email_Address is a string with a valid format, and Join_Date is stored in YYYY-MM-DD format.

3. Management & Governance

Data Governance

The policies, roles, and processes ensuring data is accurate, secure, compliant, and responsibly used.

• Example Components: Access control, compliance monitoring, stewardship.

Data Stewardship

The practice of managing and ensuring data quality within a specific domain or subject area.

• Role: Data stewards are responsible for approving changes and ensuring consistency.

Master Data Management (MDM)

A discipline focused on creating a single, consistent view of core business entities (e.g., customers, products).

• Benefit: Avoids duplication and ensures a single source of truth.

Data Quality

Measures how fit the data fits its intended purpose.

• Key Dimensions: Accuracy, completeness, consistency, timeliness.

• Impact: High-quality data leads to better analytics and decision-making.

4. Processing, Movement & Analysis

Data Integration

The process of combining data from different sources into a unified view.

• Methods: ETL, ELT, APIs, middleware.

• Goal: Make data from multiple systems work together for analysis and reporting.

• Example: Merging CRM data with financial transaction records to get a full customer view.

ETL (Extract, Transform, Load)

A data integration process where data is extracted from sources, transformed for quality and compatibility, and then loaded into a target system.

• When Used: Traditional batch processing.

ELT (Extract, Load, Transform)

A modern approach where raw data is loaded first and transformed inside the target system.

• When Used: Cloud-native warehouses like Snowflake or BigQuery.

Data Pipeline

An automated sequence of steps to move, transform, and store data between systems.

• Example: Kafka stream → Spark transformation → Redshift load.

Data Profiling

The process of analyzing data to understand its structure, quality, and relationships. It involves examining data for completeness, consistency, uniqueness, and patterns to identify anomalies or issues before processing.

• Example Output: Null count, data type mismatches, range checks.

Data Lineage

A visual or documented map showing where data came from, how it was transformed, and where it is stored.

• Value: Essential for debugging, auditing, and compliance.

Data Analysis

The process of inspecting, cleaning, transforming, and modelling data to uncover useful information, patterns, and insights that support decision-making.

• Techniques: Statistical analysis, visualization, machine learning, trend detection.

• Output: Reports, dashboards, predictive models, recommendations.

• Example: Analyzing sales trends to forecast demand, or using customer purchase history to recommend products.

5. Modern Architectural Approaches

Data Mesh

A decentralized data architecture paradigm that treats data as a product, with domain-specific teams owning and managing their data.

• Benefit: Improves scalability and reduces bottlenecks in large organizations.

Data Fabric

An architecture that uses metadata-driven automation to integrate, manage, and provide access to data across hybrid and multi-cloud environments. It simplifies data management by automating tasks like integration and governance.

• Benefit: Automation, AI-driven insights, cross-platform compatibility.

Conclusion

The data world moves fast; today’s buzzword can become tomorrow’s standard. Understanding these terms isn’t just about keeping up with industry jargon; it’s about being able to design, build, and manage systems effectively.

When I first moved from backend development into data engineering, learning this vocabulary was my survival kit. Hopefully, this guide will be yours too.

When I worked at Intuit, my team was responsible for building the data pipeline that processed TurboTax e-filing data received from the TurboTax website. This data needed to be made available for various dashboards and analytics use cases.

A major challenge was managing PII (Personally Identifiable Information) such as Social Security Numbers (SSNs). These are highly sensitive and must be protected, but the analytics team also needed to search and filter records using these fields for legitimate business insights.

In this blog, I’ll explain how we approached encryption in our pipeline, the concepts of deterministic vs probabilistic encryption, and how local vs remote encryption workflows work in practice.

🔐 Why Encryption Was Needed

PII, such as SSNs is highly sensitive. We needed a solution that:

• Protected confidentiality in case of a data breach.

• Allowed exact match search on encrypted values for analytics queries.

• Complied with data protection regulations.

To achieve this, we used Deterministic Encryption with derived keys while storing the data in Hive.

Deterministic vs Probabilistic Encryption

🎲 Probabilistic Encryption:

• Every time you encrypt the same value with the same key, you get a different ciphertext.

• This adds randomness and makes it harder for attackers to guess values based on frequency.

• However, it’s not directly searchable, since each encryption produces different results.

Encrypt('ABC', 'AES', key1) → 'qwoeoewowe' 
Encrypt('ABC', 'AES', key1) → 'c,xcslslsd' 
Encrypt('ABC', 'AES', key1) → 'fjkdfdfd;xsd'  

Encrypt('DEF', 'AES', key1) → 'fdf2424;SER' 
Encrypt('DEF', 'AES', key1) → 'kl,sek8457'

Use Case:

• Ideal when data confidentiality is the top priority and searchability is not required.

Strength:

• Defends against ciphertext pattern attacks.

Downside:

• Equality checks or searching on encrypted values are impossible without decryption.

✅ Deterministic Encryption:

• Every time you encrypt the same value with the same key, you get the same ciphertext.

• This makes it searchable because identical inputs always produce identical outputs.

• The trade-off is slightly lower security compared to probabilistic encryption, since patterns can be detected.

Encrypt('ABC', 'AES-SIV', key2) → 'adsfffdfd'
Encrypt('ABC', 'AES-SIV', key2) → 'adsfffdfd'
Encrypt('ABC', 'AES-SIV', key2) → 'adsfffdfd'

Encrypt('DEF', 'AES-SIV', key2) → 'als34asaual'
Encrypt('DEF', 'AES-SIV', key2) → 'als34asaual'

Use Case:

• Useful when you need to query encrypted fields, e.g., searching for users with a specific SSN or email.

Downside:

• Vulnerable to pattern analysis — if two ciphertexts match, they came from the same plaintext.

In our pipeline, we chose deterministic encryption because the Analytics team needed to query encrypted SSNs directly without decrypting them.

For example, if we are searching for SSN: '111-22-3333', and we know all SSNs are encrypted with key 'key2', then we can look for that SSN even if it's encrypted in the DB as

SELECT * FROM TABLE 
WHERE SSN = Encrypt('111-22-3333', 'AES-SIV', key2);

🧩 Base Keys and Derived Keys

🔑 Base Key:

A base key (master key) is the root key stored securely (e.g., in a Key Management System or HSM). It is never used directly for encrypting data.

🧪 Derived Key:

A derived key is generated from the base key using a deterministic function like HKDF to produce unique keys for specific data sets or purposes. Derived Keys are

• Are unique per user or context

• Enable key separation

• Allow rotating individual keys without touching the base key

This separation means that even if a derived key is compromised, the base key remains safe, and other derived keys are unaffected.

🏠 Local vs 🌐 Remote Encryption

🏠 Local Encryption (Client-side):

• The application server retrieves the encryption key from a secure key store (like AWS KMS, Google Cloud KMS, Azure Key Vault) and performs encryption locally.

• Pros: Faster, fewer network calls after key retrieval.

• Cons: Keys are briefly present on the application server (higher risk if server is compromised).

🌐 Remote Encryption (Server-side):

• The application server never sees the key. It sends the value to encrypt to the KMS, which returns the encrypted value.

• Pros: More secure since keys never leave the KMS.

• Cons: Slower due to extra network calls.

Glossary

• AES (Advanced Encryption Standard): A widely used symmetric encryption algorithm.

• AES-SIV (AES–Synthetic IV): A deterministic encryption mode that produces the same ciphertext for the same input and key, while still ensuring authenticity.

• HKDF (HMAC-based Key Derivation Function): A standard method to derive multiple keys from a single master key.

• KMS (Key Management Service): A secure service to store, manage, and use encryption keys (e.g., AWS KMS, Google Cloud KMS, Azure Key Vault).

Conclusion

By combining deterministic encryption with derived keys, we ensured that:

• PII was protected at rest.

• Data could still be searched for analytics.

• Encryption keys were managed securely via a base–derived key architecture.

Whether you choose local or remote encryption depends on your security requirements and performance needs. In high-security environments, remote encryption is preferred, while local encryption offers faster performance when the risk is lower.

Medallion Architecture, often visualized as the bronze, silver, and gold layers of a data platform, has become a modern standard for structuring data lakes and lakehouses. But like any architectural choice, it isn’t a universal solution.

In this blog, we’ll explore what Medallion Architecture offers, when it shines, the trade-offs it brings, and best practices to help you decide if it’s right for your organization.

What is Medallion Architecture?

Medallion architecture is a layered design pattern for organizing data in a lakehouse. It enables us to systematically improve the quality, reliability, and accessibility of data through a series of logical zones—Bronze, Silver, and Gold—each with a clear purpose:

Image from Databricks.com

This separation helps data teams manage complexity, improve data quality, and keep raw data traceable for audits and reprocessing.

• Bronze: This is the raw landing zone. Here, we directly dump data from various sources, including APIs, event logs, batch files, and streaming data. No transformations, just as-it-comes—think append-only, immutable storage. This zone provides our data lineage and audit trail.

• Silver: Here’s where the real work happens. We clean up, validate, drop corrupt records, deduplicate, standardize formats (e.g., consistent timestamps), and join across sources. This is typically the richest layer for data scientists, who require up-to-date, reliable data for modelling.

• Gold: Optimized for business and analytics consumption. These are the datasets behind our dashboards, quarterly reports, and executive summaries. We aggregate, calculate KPIs, and model according to business needs, often using dimensional models like star schemas.

When to use the Medallion Architecture

The Medallion Architecture is well-suited for specific scenarios:

1. Complex Data Pipelines:

   • When dealing with diverse data sources (e.g., streaming, batch, structured, unstructured), the layered approach simplifies ingestion, transformation, and consumption.

2. Large-Scale Data Processing:

   • Organizations with petabyte-scale data benefit from the architecture’s scalability and ability to leverage cloud storage and compute.

3. Data Quality and Governance Needs:

   • When data quality, lineage, and auditability are critical, the Medallion Architecture provides a structured framework to enforce consistency and traceability.

4. Iterative Analytics and ML:

   • Teams requiring iterative refinement of data for analytics, BI, or machine learning can leverage the Silver and Gold layers for optimized datasets.

5. Regulatory Compliance:

   • Industries like finance, healthcare, or retail that need to retain raw data for compliance benefit from the Bronze layer’s archival capabilities.

When to Avoid the Medallion Architecture

While powerful, the Medallion Architecture isn’t a one-size-fits-all solution:

1. Small-Scale or Simple Workflows:

   • For small datasets or simple ETL processes, the overhead of managing three layers may outweigh the benefits. A single-layer pipeline or traditional data warehouse might suffice.

2. Low Data Variety:

   • If data sources are uniform and require minimal transformation, a simpler architecture (e.g., direct loading into a data warehouse) may be more efficient.

3. Limited Resources:

   • Organizations with constrained budgets or expertise may struggle with the complexity of implementing and maintaining the architecture.

4. Real-Time Processing with Minimal Latency:

   • For use cases requiring ultra-low latency (e.g., real-time fraud detection), the multi-layer processing may introduce unacceptable delays unless optimized with streaming frameworks.

Benefits of using the Medallion Architecture

1. Better Data Quality

   1. Data transformations happen gradually, allowing validation rules, schema enforcement, and anomaly detection in the silver layer before data becomes business critical.

2. Reproducibility and Traceability

   1. By keeping bronze data as an immutable archive, you can replay pipelines or fix issues downstream without data loss.

3. Decoupling

   1. Different teams can consume data from bronze (for exploration), silver (for operational reporting), or gold (for executive dashboards) without stepping on each other’s pipelines.

4. Supports Multiple Use Cases

   1. It fits both batch and near-real-time ingestion patterns, making it easier to evolve your data platform as business needs change.

Trade-offs: What You Should Know

• Increased Complexity:

  • Managing multiple layers requires robust pipeline orchestration, monitoring, and governance, increasing operational overhead.

• Higher Costs:

  • Storing data across multiple layers (especially in Bronze and Silver) can increase storage and compute costs, particularly without proper lifecycle management.

• Latency:

  • The multi-layer processing can introduce latency, making it less suitable for real-time applications unless optimized with streaming tools like Apache Spark or Delta Live Tables.

• Maintenance Overhead:

  • Schema evolution, data drift, or changes in business logic require updates across layers, which can be time-consuming.

• Skill Requirements:

  • Implementing the architecture effectively requires expertise in distributed systems, data engineering, and cloud platforms.

Example: In a real-time use case where you need data in dashboards within seconds, having three separate transformations might introduce too much latency.

Archiving Decisions in the Medallion Architecture

How the Medallion Architecture Enhances Data Quality

Data quality is a cornerstone of effective analytics, and the Medallion Architecture provides a structured approach to ensure high-quality data:

1. Bronze Layer: Data Integrity:

   • By ingesting raw data without transformation, the Bronze layer preserves the original data’s integrity.

   • Schema-on-read and flexible storage formats (e.g., Delta, Parquet) allow validation of data types and structures during ingestion.

   • Archiving raw data ensures a fallback for reprocessing if quality issues arise later.

2. Silver Layer: Cleansing and Validation:

   • The Silver layer enforces data quality through transformations like:

     • Deduplication: Removing duplicate records to ensure consistency.

     • Data Validation: Applying rules to check for missing values, outliers, or invalid formats.

     • Normalization: Standardizing formats (e.g., dates, currencies) for consistency.

     • Enrichment: Joining with reference data to enhance completeness.

   • Quality checks (e.g., using tools like Great Expectations or Delta Live Tables) can be integrated to flag or quarantine problematic data.

   • Data lineage is maintained, allowing traceability to identify where quality issues originate.

3. Gold Layer: Business-Ready Data:

   • The Gold layer aggregates and curates data for specific use cases, ensuring it meets business requirements (e.g., accuracy for financial reporting).

   • Aggregations and joins reduce noise, improving reliability for analytics and ML.

   • Governance policies (e.g., access controls, data masking) ensure compliance and protect sensitive data.

4. End-to-End Quality Benefits:

   • Progressive Refinement: Each layer builds on the previous one, incrementally improving quality through validation, cleansing, and curation.

   • Reusability: High-quality Silver and Gold datasets can be reused across teams, reducing redundant processing and errors.

   • Auditability: The layered approach supports tracking data quality issues back to their source, facilitating root-cause analysis.

   • Consistency: Standardized transformations in the Silver layer ensure consistent data for all downstream consumers.

Conclusion

Medallion Architecture is a powerful, proven design pattern that can elevate your data quality, governance, and reproducibility. But it also comes with cost and complexity, so it isn’t for everyone.

If you need to scale your data platform, serve many consumers, or enforce data contracts, it’s likely a great fit. But for simpler or low-latency workloads, a leaner approach might work better.

What is a HashMap?

Imagine a library catalog where books are organized by a unique code. You provide the code, and the system quickly directs you to the exact shelf and slot where your book resides.

A HashMap works similarly; it’s a data structure that stores data as key-value pairs, using a process called hashing to map keys to specific locations in an underlying array, known as the bucket array. Each position in this array is referred to as a bucket, and multiple entries can end up in the same bucket due to collisions, which are handled using linked lists. HashMaps’ search, insert, and delete operations are extremely efficient, with an average O(1) time complexity.

The put Method: Storing Data in a HashMap

The put method is how we insert or update a key-value pair in a HashMap. Calling map.put("key", "value") triggers a series of steps that ensure the data is stored efficiently, even when collisions occur. Here’s how it works, as illustrated in the flowchart:

Step 1: Calculate the Hash Code

• What happens: The process starts by computing the hash code of the key using "key".hashCode(). This method, built into most programming languages (like Java), generates an integer that represents the key.

• Why it matters: The hash code is the first step in determining where the key-value pair will live in the HashMap.

Step 2: Find the Bucket Index

• What happens: The hash code is used to calculate the bucket index with the formula:
  index = hashCode(key) & (size - 1)
  Here, size is the number of buckets (default is 16 in many implementations), and & is a bitwise AND operation. For a size of 16, this becomes hashCode(key) & 15, ensuring the index falls between 0 and 15.

• Why it matters: This step pinpoints the bucket where the key-value pair should be stored, keeping the mapping consistent.

Step 3: Check for Hash Collisions

• What happens: The HashMap checks if the bucket at the calculated index is already occupied. This is a decision point:

  • No collision: The bucket is empty.

  • Collision: The bucket contains one or more entries (a linked list).

• Why it matters: Collisions happen when different keys produce the same hash code or index. The HashMap must handle this gracefully.

If No Collision (Step 4)

• What happens: If the bucket is empty, the key-value pair is added as the first node in a new linked list for that bucket. The process ends here.

• Why it matters: This is the simplest scenario, making insertion quick and straightforward.

If Collision Occurs (Step 5)

• What happens: If the bucket has a linked list, the HashMap checks if the key already exists by comparing it to each node’s key using "key".equals(existingKey).

  • Key exists: Move to Step 7.

  • Key doesn’t exist: Move to Step 6.

• Why it matters: This ensures the HashMap either updates an existing entry or adds a new one without duplicates.

Key Doesn’t Exist (Step 6)

• What happens: If no matching key is found, the key-value pair is added as the next node in the linked list.

• Why it matters: The linked list grows to accommodate multiple entries in the same bucket, resolving the collision.

Key Exists (Step 7)

• What happens: If the key is found, the existing value is replaced with the new value provided in put.

• Why it matters: This allows the HashMap to update values for existing keys efficiently.

Example in Java

HashMap<String, String> map = new HashMap<>(); 
map.put("apple", "red");  // New entry in bucket 
map.put("banana", "yellow"); // New entry, possible collision map.put("apple", "green"); // Updates "apple" to "green"

The get Method: Retrieving Data from a HashMap

The get method retrieves the value associated with a key, as in map.get("key"). It’s a search operation that mirrors parts of put but focuses on finding and returning data. Let’s break it down:

Step 1: Calculate the Hash Code

• What happens: Just like put, it starts with "key".hashCode() to get the hash code.

• Why it matters: The hash code identifies the potential location of the key.

Step 2: Find the Bucket Index

• What happens: The bucket index is computed using hashCode(key) & (size - 1).

• Why it matters: This directs the HashMap to the correct bucket to begin the search.

Step 3: Check the First Node

• What happens: The HashMap compares the key with the first node’s key in the bucket’s linked list using "key".equals(firstNodeKey).

• Why it matters: This is the entry point to the bucket’s data.

Step 4: Are the Keys Equal?

• What happens: If the keys match, move to Step 5. If not, proceed to Step 6.

• Why it matters: This determines whether we’ve found the target key immediately or need to keep looking.

Key Found (Step 5)

• What happens: If the keys match, the value from that node is returned, and the process ends.

• Why it matters: This is the goal—fast retrieval of the value.

Check More Nodes (Step 6)

• What happens: If the keys don’t match, the HashMap checks if there’s another node in the linked list.

  • Yes: Loop back to Step 4 for the next node.

  • No: Move to Step 7.

• Why it matters: The linked list must be traversed fully to ensure the key isn’t missed.

Key Not Found (Step 7)

• What happens: If no matching key is found after checking all nodes, null is returned.

• Why it matters: This handles cases where the key doesn’t exist in the HashMap.

Example in Java

HashMap<String, String> map = new HashMap<>();
map.put("apple", "red");
System.out.println(map.get("apple")); // Outputs: "red"
System.out.println(map.get("orange")); // Outputs: null

Key Takeaways

• Efficient Lookups: HashMap offers near-constant time operations for get and put, leveraging the hash code mechanism.

• Collisions Handled Gracefully: When collisions occur, HashMap uses a linked list structure within buckets to store multiple items.

• Overrides and Insertions: The put method either inserts new entries or replaces existing ones if the key already exists.

Understanding these mechanisms helps write performant, bug-free code, and clarifies why certain operations, like having good hashCode and equals Implementations are essential for objects used as keys in a HashMap.

In large-scale data environments, query performance and scalability hinge on how well operations can be parallelized. Key SQL clauses such as ORDER BY and window functions, especially those combining PARTITION BY and ORDER BY, play a central role in determining parallel processing efficiency across big data platforms. This blog explores how ORDER BY works, its role in window functions, and strategies to mitigate its effect on distributed processing.

What is ORDER BY?

The ORDER BY clause sorts a query’s result set by one or more columns, either ascending (ASC) or descending (DESC). In Big Data, this is a global operation requiring all data to be sorted across the cluster.

For Example:

SELECT * FROM employees ORDER BY salary DESC;

This sorts the entire dataset by salary, shuffling data to a single reducer, which limits parallelism.

Window Functions with ORDER BY

Window functions perform calculations across a set of rows (a "window") related to the current row. The PARTITION BY clause groups data into partitions, and ORDER BY sorts rows within each partition.

For Example:

SELECT
  employee_id,
  department_id,
  salary,
  RANK() OVER (PARTITION BY department_id ORDER BY salary DESC) AS rank_in_dept
FROM employees;

Here, PARTITION BY department_id groups rows by department, and ORDER BY salary DESC sorts within each partition to assign ranks.

Key Components of Window Functions

• PARTITION BY: Divides the dataset into partitions, allowing parallel processing within each partition.

• ORDER BY: Specifies the sorting order within each partition for the window function.

• Window Function: Examples include ROW_NUMBER(), RANK(), DENSE_RANK(), SUM(), AVG(), etc.

Impact on Parallelism

In distributed systems, parallelism enables scalability by processing data across multiple nodes. However, ORDER BY can reduce parallelism:

1. Global ORDER BY:

   • Requires shuffling all data to a single reducer for sorting.

   • Eliminates parallelism, creating a bottleneck as all rows are funneled to one node.

2. ORDER BY with PARTITION BY:

   • Partitions allow parallel processing, as each partition is handled independently.

   • Sorting within partitions still requires computational effort, and skewed partitions (uneven sizes) can overload some nodes, reducing parallelism.

Example:

SELECT
  employee_id,
  department_id,
  salary,
  SUM(salary) OVER (PARTITION BY department_id ORDER BY salary) AS running_total
FROM employees;

This query computes a running total within each department, sorted by salary. Partitions are processed in parallel, but sorting within large or skewed partitions adds overhead.

Practical Example in Apache Spark

Let’s illustrate with a Spark example. Suppose you have a dataset of sales transactions:

from pyspark.sql import SparkSession
from pyspark.sql.window import Window
from pyspark.sql.functions import sum, col

# Initialize Spark session
spark = SparkSession.builder.appName("WindowFunctionExample").getOrCreate()

# Sample data
data = [
    (1, "Electronics", 1000),
    (2, "Electronics", 1500),
    (3, "Clothing", 500),
    (4, "Clothing", 700),
    (5, "Electronics", 2000)
]
df = spark.createDataFrame(data, ["sale_id", "department", "amount"])

# Define window specification
window_spec = Window.partitionBy("department").orderBy(col("amount").desc())

# Calculate running total within each department
df_with_running_total = df.withColumn(
    "running_total",
    sum("amount").over(window_spec)
)

# Show results
df_with_running_total.show()

Output:

+-------+-----------+------+-------------+
|sale_id|department |amount|running_total|
+-------+-----------+------+-------------+
|    5  |Electronics|  2000|        2000 |
|    2  |Electronics|  1500|        3500 |
|    1  |Electronics|  1000|        4500 |
|    4  |Clothing   |   700|         700 |
|    3  |Clothing   |   500|        1200 |
+-------+-----------+------+-------------+

Analysis

• Partitioning: The partitionBy("department") ensures that data is grouped by department, allowing parallel processing of "Electronics" and "Clothing" partitions.

• Sorting: The orderBy(col("amount").desc()) sorts rows within each partition, which is less resource-intensive than a global sort.

• Parallelism: Spark processes each partition independently, but sorting within partitions still requires computational resources. If the "Electronics" partition were much larger, it could slow down the overall job.

Best Practices for Using ORDER BY in Big Data

1. Minimize Global ORDER BY: Avoid global ORDER BY unless absolutely necessary. Use PARTITION BY to limit sorting to smaller subsets.

2. Monitor Data Skew: Use tools like Spark’s UI to check partition sizes and ensure even distribution.

3. Tune Spark Configurations: Adjust spark.sql.shuffle.partitions to control the number of partitions during shuffling, balancing parallelism and overhead.

4. Profile Queries: Use query execution plans (e.g., EXPLAIN in Spark or Hive) to identify bottlenecks caused by sorting or shuffling.

5. Limit Sorted Data: Apply filters (e.g., WHERE clauses) to reduce the dataset size before sorting.

6. Optimize Data Distribution: Ensure data is pre-partitioned or distributed evenly to avoid skew. For example, in Spark, use repartition() or distribute by to balance data before applying window functions.

7. Use Appropriate Hardware: Increase the number of nodes or cores to handle sorting tasks in parallel, especially for large datasets.

Conclusion

ORDER BY in Big Data reduces parallelism due to shuffling and sorting, especially in global operations. Using PARTITION BY in window functions helps by enabling parallel processing within partitions, but sorting overhead and data skew can still impact performance. Optimize with careful partitioning, filtering, and tuning to balance functionality and scalability.