
Big Data Through a System Design Lens — Part 1
A database grows to 1TB.
Does that mean it's time to introduce Apache Spark, Kafka, or a distributed database?
Not necessarily.
One of the important distinctions in System Design is understanding the difference between having a large amount of data and having a data processing problem that requires a different architecture.
- Data Size Is Not the Entire Story
Consider two hypothetical systems, each storing 1TB of data.
System A
Processes analytical reports in 5 minutes.
Business requirements allow up to 60 minutes.
Meets its current performance objectives.
System B
Takes 10 hours to generate reports.
Business requirements demand results within minutes.
Cannot meet its performance objectives.
Both systems have the same data volume, but very different engineering challenges.
Even with the same dataset size, performance can vary significantly based on workload, indexes, data models, hardware, concurrency, and query design.
The architecture should respond to those constraints rather than the storage number alone.
- Understanding the Three Vs of Big Data
Big Data is often described through three characteristics:
Volume
How much data must the system store and process?
Important questions include:
Can the existing infrastructure handle projected data growth?
Are storage or query costs increasing?
Can processing complete within the required time?
Velocity
How quickly does data arrive, and how soon must it be processed?
For example, a system collecting millions of events may need continuous ingestion and near-real-time processing.
The challenge is not necessarily how much data exists, but whether processing can keep up with incoming traffic.
Variety
What types of data must the system handle?
A system may combine structured transactions, JSON events, application logs, images, and IoT data.
Different data formats and access patterns can introduce ingestion, storage, and processing complexity.
These three dimensions help explain Big Data challenges, but they don't establish a universal numerical threshold.
- Before Choosing Distributed Architecture
Suppose database queries become slower as the dataset grows.
Should we immediately introduce distributed processing?
Usually, the first step should be investigating the actual bottleneck.
Step 1: Define the requirements
Identify:
Expected data growth
Query latency targets, including P95/P99 where relevant
Ingestion throughput
Processing deadlines
Availability and recovery requirements
Infrastructure cost constraints
Step 2: Measure the bottlenecks
Examine:
Query execution plans
Missing or inefficient indexes
CPU, memory, and disk I/O utilization
Lock contention
Connection saturation
Processing throughput and backlog
Step 3: Evaluate architectural options
Depending on the findings, potential solutions include:
Query optimization and indexing
Improve access patterns before introducing additional infrastructure.
Partitioning
Divide large datasets into manageable partitions. Partitioning can happen within a single database or across distributed storage nodes, depending on the design.
Asynchronous processing
Move non-immediate tasks into background processing pipelines to decouple ingestion from computation.
Horizontal scaling
Distribute suitable workloads across multiple servers when a single machine cannot meet the required capacity.
Distributed processing
Use multiple machines to process datasets in parallel when the workload and performance requirements justify the additional complexity.
These techniques solve different problems. They are not interchangeable.
- Does Every Big Database Need Kafka or Spark?
No.
Apache Kafka is designed for distributed event streaming and is useful for workloads that benefit from durable event ingestion and independent consumers.
Apache Spark is a distributed processing engine used for large-scale data transformations and analytics.
Neither is an automatic solution to a slow database query.
Selecting either technology without understanding the workload can introduce unnecessary operational overhead.
- The Engineering Decision
Before changing the architecture, ask:
What specific requirement are we failing to meet?
What evidence identifies the bottleneck?
Can we improve the existing design?
What trade-offs will the new architecture introduce?
How will we measure whether the change succeeded?
Final Thoughts
Big Data is not defined by a single database size.
Its challenges emerge from the relationship between data characteristics, processing requirements, and the capabilities of the system.
Good System Design starts with understanding the problem, not selecting the technology.
This is Part 1 of Big Data Through a System Design Lens.
Next: The Big Data Pipeline — From Data Generation to Analytics Dashboards.
About the author
- Senior Software Engineer | Backend Development & Software Architecture Building scalable systems and sharing practical insights on System Design, Distributed Systems, Backend Engineering, and the engineering decisions behind reliable software.
Comments
Join the discussionNo comments yet. Be the first to comment.