It is. Until it’s every view on YouTube.

Then the same one-line operation turns into the most architecturally consequential problem in your stack.

Simple lie

Everyone starts small. For all the right reasons.

Throw it in Postgres, add an index, scale up the box if needed, sprinkle some Redis if it gets hot. Done. Like reallllyyy done.

Right up until the moment your platform goes viral. That’s kinda problem Youtube deals with, everyday.

A video goes viral and your “simple counter row” becomes the slowest object in your entire system.

Counting itself isn’t one problem. It’s a combination of 6 problems wearing a trench coat, and youtube has dealt with all of those.

The hard problems

1. Counter overflow. Sounds boring. It’s not.

“Gangnam Style” hit 2^31 views and crashed YouTube’s display because the field was a signed 32-bit integer. They migrated to 64-bit and bought themselves a few quintillion views of headroom.

But, relatively easy gotcha.

2. Volume & Concurrency. A million events per second means a million writes per second.

YouTube serves a million video views per second. But raw volume isn’t even the worst part. Concurrency is.

If you use postgres, every increment on a hot counter takes a row lock. Postgres uses MVCC, so an UPDATE is actually an insert plus a tombstone, meaning your hot row is generating thousands of dead versions per second that vacuum has to chase down.

The row lock serializes everyone. The fsync caps you again.

You’ve got a 96-core machine watching one logical row’s lock chain decide the throughput of your entire product. The fanciest hardware in the world bottlenecked by one number.

3. Distribution. Events happen in Tokyo. The counter lives in Iowa.

Physics is in the room, and it doesn’t care about your roadmap. You cannot make this fast if every increment needs a round trip to one global database.

4. Late events. A view happens at 09:59:55 and arrives at 10:00:55.

View events usually have streaming processor working with watermarks. That event belongs to the 9 AM bucket. The 9 AM bucket already closed.

Now what? How long do you wait? Do you process that late event? What about the event that shows up tomorrow?

5. Exactness is contested. Did the user actually watch, or click and bounce? Bot? Self-view? Same person refreshing?

6. Failures. A consumer crashes mid-batch. A network splits. Did the event get counted? Twice? Lost? Pick: at-most-once, at-least-once, exactly-once. Each one wrecks your architecture in a different direction.

7. Hot keys: no matter how you shard, one celebrity’s post will overwhelm one physical row.

Youtube uses BigTable

Of course, YouTube doesn’t use Postgres for view counts. They use Bigtable.

The mental model:

LSM tree = append to log + insert into memtable
no row locks, no in-place updates, no vacuum

Every Bigtable write is just an append to a commit log plus an in-memory insert into a sorted structure.

No random disk write. No lock that serializes writers to the same row; writes are timestamped and merged later.

When a single row gets too hot, the tablet splits automatically and the load redistributes.

A single Bigtable cluster eats millions of writes per second without sweating. The view counter you see ticking up under a viral video? That’s a IncrementColumn on one row, absorbed at memory speed.

But Bigtable can’t tell you the top 100 most-viewed videos. There’s no ORDER BY view_count DESC LIMIT 100. No secondary indexes by default.

That’s a whole different topic. We might cover it separately.

But..do we even need to count precisely?

Sometimes the right count is wrong approximate count.

Reddit uses it. Google Analytics uses it. The number you see is statistically a lie (close enough though). It’s also correct enough that nobody cares.

Count-Min Sketch does the trick for frequency counts and approximation.

YouTube actually utilizes the Count-Min Sketch algorithm to track real-time view frequencies, trending topics, and to identify the "Top K" most viewed across its massive global traffic.

Count-Min Sketch could be visualised is a tiny 2D grid of counters (say 5 rows × 1000 columns) paired with one hash function per row. Play with it here.

To record an item, you hash it through each function and bump the cell it lands on in every row: 5 hashes, 5 increments, done.

To ask “how many times have I seen this?”, you hash it again, look up those same 5 cells, and return the minimum value. Why min?

Because different items can collide on the same cell and inflate it, but they almost never collide in all rows, so the smallest cell is your cleanest read.

You get approximate frequency counts for billions of items in a few kilobytes of memory, the estimate is always an upper bound (it lies high, never low), and two sketches merge by just adding their cells together which is why every distributed streaming system reaches for it.

Summary

This is the mental model: counting accuracy is a knob, not a constant. Pick it per access pattern and distribution.

The dashboard showing "1.2M views" can be a HyperLogLog estimate that's off by 1% and nobody cares.

But if you are not viral, then a single tick from 99 views to 100 views may matter to you.

Think differently when designing your systems.

Well, yes and no.

Sure, AI and Machine Learning (ML) are big parts of it, but they rely on data—lots of data (Big Data, anyone?)—to train model and be analyzed.

Humans like Data Analysts and Scientists spot trends in the market, then tweak strategies to make sure the recommendation systems are on point.

But it’s Data engineering and engineers that makes the magic happen. Huge Respect!

But hey, I’m a software engineer, so why should I care about data engineering? 🧐

Think twice, you might be wrong!

Today, I'll give software engineers a quick intro to data engineering, covering:

• Why It Matters: Why learning data engineering basics is helpful, even if you’re not aiming to become a data engineer.

• Role Breakdown: A simple example to explain the roles of software engineers, data engineers, analysts, and scientists, and how they work together.

• Tech Overview: A snapshot of data engineering platforms, tech stacks, and processing workflows.

👋 Hey there, I am Gourav. I write about Engineering, Productivity, Thought Leadership, and the Mysteries of the mind!

Why should software engineers care about data engineering?

As a junior engineer, terms like "Big Data," "Spark," and "Data Warehouse" would fly right over my head.

I used to think, "That’s for the data engineers. I’m busy building distributed systems, so why should I bother?"

Being a senior engineer, I realized knowing data pipelines, batch-processing, and data analysis influences my system’s design.

So, I rolled up my sleeves and dove into a project and got hands-on experience with MapReduce, and Spark—just enough to get by.

Then, I got pulled into other business-critical projects and that little spark (pun intended) of interest in data engineering fizzled out.

As a staff engineer now, there’s no more room for excuses. I need to understand enough to help the company make tradeoff decisions.

In general, software engineers benefit by just knowing the basics:

• Better System Design & Architecture: Knowing data pipelines will help you understand their trade-offs. You'll build more scalable, efficient systems.

• Enhanced Debugging: Ever had a bug that’s tied to data? Understanding data will make you a better problem-solver when those tricky issues pop up.

• Data-Informed Decision Making: As you grow into leadership roles, understanding data engineering will help you. It will give you more insight into product, system, and business decisions.

The Difference Between Software Engineers, Data Engineers, Data Analysts, and Data Scientists

Here’s the breakdown:

• Data Engineers build data pipelines—systems that transform raw data into something useful for analysis.

• Data Analysts take that clean, processed data and use it to find patterns and trends, helping companies figure out what’s happening now.

• Data Scientists go a step further by building models that predict future trends and offer recommendations.

• Software Engineers build the systems that solve the problems those data insights reveal.

In short, the data engineer sets up the infrastructure. The data analyst finds current trends. The data scientist makes future predictions. The software engineer builds the product that uses them. 🚀

Data Engineering Tech Stack: A Primer for Software Engineers

Oh man! The field’s come a long way, and it’s still changing fast!

Concepts like Data Lakes and Lakehouses and tools like Trino, Flink, data formats like Apache Iceberg, Avro, and Parquet.

More confusing than ever!

I’ll try to explain simply and cover only the Tech Stack of Data Engineering.

Tech Stack of Data Engineering

The image below is a courtesy of . Thanks to him for creating this.

As a software engineer, you're likely familiar with the request lifecycle in an app. It starts with an API call (ingestion), runs some business logic (processing), then saves it to a database (storage). This is what we call Online Transaction Processing (OLTP), where the system handles day-to-day operations.

Data engineering works similarly but picks up after your application’s database. Just as APIs bring in data for apps, data engineers create pipelines. They process and store data on a much larger scale for analytics. This is called Online Analytical Processing (OLAP).

The Basics: ETL vs. ELT

Data engineering revolves around one key idea: building systems that ingest, process, and store data in ways that make it useful for analysis. This process is called building a data pipeline, and the most common architecture for this is ETL:

• Extract: You grab raw data from various sources like APIs, databases, or files.

• Transform: You clean and shape the data into a usable format.

• Load: You store it somewhere (like a data warehouse) where analysts and business tools can easily access it.

There’s also ELT, which is like ETL’s cooler cousin. Instead of transforming the data right away, you load it into a storage system first and then transform it as needed. This is especially useful when you don’t know upfront what the data might be used for. It keeps your options open and flexible.

Data Warehouses, Lakes, and Lakehouses

Where does all this data go? Great question!

• Data Warehouse: This is structured storage for quick analysis. It’s fast and optimized for answering specific questions.

• Data Lake: This is a giant storage pool for raw data. It’s great for when you have tons of different data but aren’t sure yet what you want to do with it.

• Lakehouse: A happy blend of the two. It offers the flexibility of a data lake with the structured query power of a data warehouse.

The evolution from warehouses to lakes to lakehouses happened because businesses needed faster, more scalable ways to handle growing amounts of data.

Big Data: The Heavyweights

Big Data is all about handling massive amounts of information, the kind that Amazon, Netflix, and Instagram deal with every day. Traditional systems can’t handle this scale, so we use distributed systems like Hadoop.

Hadoop is an ecosystem that chops up huge datasets and processes them across multiple servers at once, kind of like an army of worker bees.

Here’s the core of Hadoop:

• HDFS (Hadoop Distributed File System): Stores huge amounts of data.

• MapReduce: Breaks down data tasks and processes them in parallel.

• YARN: Keeps everything running smoothly.

Other cool tools you should know about:

• Apache Spark: A super fast engine for processing big data in near real-time.

• Apache Kafka: A messaging system for streaming data.

• Apache Flink: A tool that handles both real-time and batch processing.

These tools are like the superheroes of Big Data, ensuring massive amounts of data can be processed and analyzed efficiently.

File Formats: Keeping Data Organized

When managing massive amounts of data, the format you choose can make or break performance and flexibility. Let’s break down some popular data formats you’ll see in data warehouses and lakehouses:

Iceberg & Delta (Lakehouse Formats)

• Apache Iceberg is the table format for huge, distributed datasets. It supports file formats like Parquet and ORC, and cool features like time travel (yes, you can query old data!) and dynamic partitions. Works great with Spark, Trino, and Flink.

• Delta Lake (by Databricks) is a fast, real-time data pro, optimized for Spark. It supports Parquet, gives you schema control, and speeds up read-heavy tasks like machine learning.

JSON & Parquet (Warehouse Formats)

• JSON is the human-friendly, semi-structured format perfect for web apps, but not the fastest for big data crunching.

• Parquet is the warehouse star! A columnar format is perfect for large datasets. It's efficient for analytics and supported by all the big data platforms.

There’s a lot about this data format that we can detail out but that’s beyond the scope of a basic primer. I may cover this in a future edition of Tech Comprehension.

Wanna build a strong Data Engineering understanding and make a career in it?

If you want to build a solid understanding of Data Engineering, try out ’s bootcamp here. Use coupon code “GOURAV” and you will get 25% off. :)

Shoutouts 🔊

A big shout-out to all the Data Engineering folks I know , , , . You all are rockstars!

Also, special shoutout to other Tech newsletter writers - , , - writing tech deep dive newsletter is hard.

Interesting reads last week:

1. SWE v/s MLE by and

2. 5 Keys to the Hiring Manager Interview from a Meta Senior Manager by and

3. Being an engineering manager at Amazon by and

4. I Worked More and Achieved Less by

🤝 Let’s Connect

Sponsorship | Collaboration | LinkedIn | 1:1 Mentoring | Twitter

Gourav Khanijoe

👋 Hey there, fellow tech enthusiasts, professionals, and wanderers of the digital realm! 🚀

I am Gourav. I write about Engineering, Productivity, Thought Leadership, and the Mysteries of the mind!

Welcome back and Happy New Year!

(I clicked that pic on the New Year celebration at Space Needle, Seattle, Washington!)

Today, I am going to dive into an interesting puzzle of how many software systems crashed simultaneously; share a thought-provoking story, and expose what I am addicted to!

Your engagement is what makes this community vibrant, so feel free to hit reply and share your thoughts, feedback, and ideas, or even just to say hello!

Today at a Glance

• Survey: It’s not your fault!

• A Thought-provoking Quote

• Main Topic: A ‘time’ when multiple Software Systems crashed around the world

• What am I addicted to?

Survey: It’s not your fault!

Despite having so many resources to learn, cracking a new skill or gathering life-changing knowledge isn’t simple.

It's not your fault that traditional learning methods demand time we don't have. Books are a time-consuming voyage, and audiobooks lack that personal touch.

Now, there are two main ways to learn.

1/ Learn by making mistakes yourselves;

2/ Learn through the wisdom gleaned from the mistakes of others

Fill out this free survey in under 30 seconds, to help me understand the topic that you would like to learn and I will share my long-gathered wisdom via a course!

As a token of appreciation, you will enjoy an exclusive 20% discount when our course flings open its doors to enrollment.

Seize the opportunity to transform your future—let's make your career goals a reality! 🚀

Link to the Survey

Thought-provoking Quote

This interesting story would help you retain this deeply:

A merchant traveling with his three camels stopped at an inn at night and had only two ropes to tie them to the tavern.

After tying two camels, as he fell short of the rope, he got a brilliant idea.

He simply pretended to put a noose around the third camel’s neck and then attach that virtual rope to a peg.

Thinking it was tied, the third camel sat down timidly, like the others.

In the morning, when they had to leave, he untied two camels, but the third one refused to budge…

…because it was under the impression that it was also tied, the merchant had to do a virtual or fake untie to make the third camel move.

Like the camel, many times, we involuntarily tie ourselves to false beliefs created by our own minds.

We must, therefore, crush those briefs like “I am a failure”, and “Success isn’t for me” and rather train our minds to think positively.

A story about a ‘time’ when many Software Systems around the world crashed at the same time…

On the night of June 30th to July 1st, 2012, many online services and systems around the world crashed simultaneously.

The servers locked up and stopped responding.

Many airlines could not process any reservations or online check-in for several hours.

So, what happened here?

Well, A Leap second happened.

Source of picture: https://in.pinterest.com/pin/742249582348637613/

Before we can understand what happened, we need to understand what is leap second.

In short,

Leap seconds are inserted in UTC to keep UTC and GMT from drifting apart.

GMT, UTC…leap seconds…what the heck…

what is even a ‘second’? How is time calculated?

Explain ‘Time’ like I’m a novice!

Don’t worry, let’s understand it in a very simple manner.

The notion of time is related to the duration it takes for Earth to revolve around the sun.

GMT: stands for Greenwich Mean Time, the mean solar time at the Royal Observatory in Greenwich on the south bank in Eastern London, UK. When the sun is at its highest point exactly above Greenwich, it is 12 noon GMT. Except: The Earth spins slightly unevenly, so 12 noon is defined as the annual average, the mean of when the sun is at its highest, its culmination.

So, GMT is based on astronomical observations.

Atomic Clocks and Time: This is the international way of defining time. Internal Atomic Time (TAI) is based on the caesium atom’s resonant frequency. So, put simply, we count the oscillations in this atom to calculate time.

1 day = 20 x 60 x 60 x 9,192, 631, 770 periods of caesium atom’s resonant frequency 

Thus, this is based on the Quantum mechanism and is very accurate.

The problem is our Earth 🌏 …. lol

Remember, GMT is based on astronomical observations. The problem is that the earth’s rotation speed is not constant. Earth’s rotation slows down or increases because of earthquakes and tidal friction.

So, the 1-second duration may sometimes differ in GMT.

Thus, GMT is not accurate but is still widely used and the Atomic clock is accurate but could go unmatched by GMT.

So, enter UTC, as a compromise.

UTC, which stands for Coordinated Universal Time is TAI, but with corrections applied to account for Earth’s rotation (astronomy).

Leap Second: It is the adjustment applied to UTC to make it consistent with GMT. It’s an extra second that could be added to or removed from TAI to accommodate GMT. So, you may end up with a time like ‘00:59:60’ 😅

But it’s done on a certain date. Either 30th June or December 31st. And this is usually announced several months beforehand.

But back then, how were Software systems used to handle this extra or less 1-second?

Any guesses?

The answer is the systems ignored it! 😸

In many applications, 1 second doesn’t make a huge difference but in other systems, it makes quite a significant difference.

Hope you got the answer to the puzzle of what happened on June 30th, 2012. The bug in Linux kernels caused livelock on leap second, causing many internet services to go down!

Fortunately, now it’s handled by smearing or spreading that one second across a day so we don’t have to insert or remove a second explicitly.

[Source]

What am I addicted to?

Yes, I am addicted to something.

Something positive…or life-changing.

And that’s my morning routine…!

I literally crave my morning routine. When I miss it (rarely, but it happens) my whole day is affected. I don’t feel accomplished that day.

So, what’s my morning routine?

4:30-5am: wake up

5am-5:30am: drink water, prepare coffee, and enjoy that in the silence while creating a 3-4 pointer To-do list for succeeding at my day!

5:30-6:00am: meditation, or listening to thought-provoking spiritual lectures.

6-7am: “Dawn Creations: Mind and Body Boost” - I do one of these each day: exercise while listening to podcasts; newsletter content creation/researching; working on my website; or reading technical / non-fictional books.

7-7:30am: some household chores

7:30-8am: freshen up and get ready for work!

8am: start office work!

It’s because of my morning routine that this content is coming all the way to your inbox!

It’s because of the morning rituals, that I feel productive the whole day…

But how did I get into this morning's build in the first place?

That’s a topic for another day, but here’s a short answer: We have an infant at home :)

In case you missed my previous articles…

Dancing into 2024

Why do we fall short Despite Intelligence and Skill Excellence?

The Art of Empowering Others with Influential Leadership: Spreading Joy and Happiness

Dispelling Career Advice Myths in the Tech World

How to apply Bhagavad Gita teachings to succeed at Work (without Stress)?

Let’s Connect

If we haven't crossed paths previously, I'd be delighted to receive a message from you! Don't hesitate to reply to this email and introduce yourself. I make it a point to respond to every direct message, email, and comment. You can also find me actively sharing on LinkedIn—let's connect:

🤝 Connect with me on LinkedIn

𝕏 Connect with me on Twitter

💪 Connect for 1:1 mentorship

Best regards,

Gourav Khanijoe