Adding More Servers Won't Save You: Understanding Horizontal Scaling, Load Balancing & Bottlenecks

When an application starts getting more traffic, one of the first solutions engineers think about is — and this is exactly the kind of reasoning we document on engineering at Intervues:

"Let's add more servers."

It sounds logical.

If one server handles 1,000 requests per second, then surely:

1 server  → 1,000 RPS
2 servers → 2,000 RPS
10 servers → 10,000 RPS

Right?

Not necessarily.

Adding more application servers can increase capacity—but only if the rest of your system can keep up.

Your application server might be capable of handling 10,000 RPS while your database can only handle 3,000.

Your cache might become saturated.

An external API might enforce a rate limit.

A shared connection pool might run out.

And suddenly, adding more servers doesn't solve your scalability problem.

It simply moves the bottleneck somewhere else.

This is one of the most important concepts in system design and backend architecture:

You don't scale a system by adding servers. You scale the bottleneck.

Let's understand why.

1. What Is Horizontal Scaling?

Horizontal scaling, also called scale-out, means increasing system capacity by adding more instances instead of making a single machine more powerful.

Instead of:

you add multiple servers:

The idea is straightforward:

Distribute workload across multiple machines.

Horizontal scaling is one of the fundamental techniques used to build scalable web applications, APIs, backend services, and distributed systems.

But there is a major requirement.

The workload must actually be distributable.

And that's where load balancing and stateless application design become important.

2. Load Balancing

If you have multiple application servers, something needs to decide:

Which server should handle this request?

That's the job of a load balancer.

A simplified architecture looks like:

The load balancer receives incoming traffic and distributes requests across the available application servers.

Common load-balancing strategies include:

• Round robin
• Weighted round robin
• Least connections
• Least response time
• IP hashing
• Consistent hashing

The exact algorithm matters less than the fundamental idea:

Incoming requests should be distributed across available capacity.

3. Request Distribution

Imagine your application receives:

3,000 RPS

and you have three identical servers.

A simple round-robin strategy might distribute traffic like:

That's great if:

• Servers are identical
• Requests cost roughly the same
• All servers are healthy
• Dependencies can handle the combined workload

But real systems aren't always this simple.

Imagine some requests are cheap:

GET /health
GET /profile

while others are expensive:

POST /generate-report
POST /process-video
POST /checkout

Sending the same number of requests to each server doesn't necessarily mean sending the same amount of work.

This is why load balancing isn't simply:

"Split requests 50/50."

The goal is:

Distribute workload so that available capacity is used effectively.

4. Stateless Application Servers

Horizontal scaling becomes much easier when application servers are stateless.

What does stateless mean?

It means:

A server does not depend on locally stored request state to correctly handle the next request.

Consider a simple API:

If the application is stateless, any server can process any request.

That's extremely useful.

But imagine Server A stores important session information in its local memory:

Then the load balancer sends the next request to Server B.

Server B doesn't know about that session.

You now have a problem.

5. Why Statelessness Enables Scaling

Consider a stateless architecture:

The servers themselves don't own important persistent state.

Instead, state lives in shared systems such as:

• Database
• Redis
• Object storage
• Distributed cache
• Message queues

Now any server can handle any request.

If traffic increases:

3 servers → 6 servers

the new servers can immediately start processing requests.

This is one of the biggest reasons stateless architecture is so valuable for scalable web applications.

6. Stateless Does NOT Mean No State

This distinction is important.

A stateless application doesn't mean:

"The system has no state."

Obviously, most applications have state.

Users have:

• Accounts
• Orders
• Preferences
• Sessions
• Payments
• Messages
• Files

The point is:

Application instances shouldn't be the authoritative owners of state that must survive between requests.

Instead:

Now application instances can be created and destroyed without losing important state.

That makes:

• Autoscaling
• Load balancing
• Failover
• Rolling deployments
• Container orchestration

much easier.

7. The Hidden Problem: Shared Dependencies

Now we reach the real problem with:

"Just add more servers."

Consider:

You successfully scaled the application layer.

But all three servers still depend on the same database.

That's called a:

Shared dependency.

A shared dependency is a component used by multiple instances or services.

Common shared dependencies include:

• Databases
• Caches
• Message brokers
• Object storage
• External APIs
• Authentication services
• Payment gateways
• Search systems

And they can become your next bottleneck.

8. The Database Is Often the Shared Bottleneck

Let's say one application server handles:

1,000 RPS

You add five servers:

5 × 1,000
= 5,000 RPS

Sounds great.

But suppose every request generates:

5 database queries

Then:

5,000 API RPS
× 5 DB queries
= 25,000 DB queries/sec

Your application layer might be perfectly capable of handling 5,000 RPS.

But your database might not be.

The architecture is now:

Adding more application servers now makes the situation worse.

Why?

Because more servers generate even more database traffic.

9. This Is Called Bottleneck Migration

One of the most useful concepts in scalability engineering is:

Bottleneck migration.

Suppose initially:

Application CPU = 95%
Database CPU    = 40%

The application is the bottleneck.

You add more application servers.

Now:

Application CPU = 50%
Database CPU    = 95%

Congratulations.

You solved the application bottleneck.

But you created a database bottleneck.

The bottleneck migrated.

This is normal in distributed systems.

Scaling one layer can expose the next limiting component.

You can think of a system like a chain:

The system's overall capacity is constrained by its limiting components.

Improve one constraint and another may become visible.

10. Scaling the Wrong Layer

This is where many system design discussions go wrong.

Imagine your architecture is:

Someone says:

"We need more application servers."

That's probably the wrong layer to scale.

Your application layer isn't the limiting factor.

The database is.

Adding another 100 application servers would only produce:

You have spent more money without increasing meaningful system capacity.

This is why a good system designer asks:

Which component is actually limiting throughput?

before deciding:

What should we scale?

11. Cache Is Also a Shared Dependency

Caching is one of the most common techniques for improving application performance.

For example:

Instead of querying the database every time:

This can dramatically reduce database load.

But now Redis becomes a shared dependency.

Imagine:

If all ten servers depend on the same Redis cluster, Redis itself must scale with the workload.

You might eventually see:

Application CPU → 30%
Database CPU    → 30%
Redis CPU       → 95%

Your bottleneck has moved again.

Caching doesn't eliminate scalability problems.

It changes where the work happens.

12. External Services Can Become Bottlenecks

Your infrastructure isn't the only thing that needs to scale.

Imagine your application calls:

Your application can handle:

100,000 RPS

But the external service allows:

10,000 requests/minute

Your application's theoretical capacity doesn't matter.

The external service is now the constraint.

Other examples include:

• Payment gateways
• Email providers
• SMS APIs
• Maps APIs
• AI APIs
• Authentication providers
• Shipping APIs
• Third-party analytics
• Social APIs

External dependencies introduce constraints that you may not control.

These can include:

• Rate limits
• Latency
• Availability
• Quotas
• Connection limits
• Pricing

So when performing system design capacity planning, external services must be included in the analysis.

13. A Realistic Example

Let's design a simple e-commerce backend.

The architecture:

Initially:

Traffic = 3,000 RPS

Application servers:

CPU = 80%

Database:

CPU = 50%

Redis:

CPU = 40%

The application layer is the limiting component.

So we scale horizontally:

3 servers → 6 servers

Now:

Application CPU = 45%

Great.

But database load increases because there are more requests being processed.

Now:

Database CPU = 95%

The database becomes the bottleneck.

We scale or optimize the database.

Perhaps we introduce:

• Better indexes
• Query optimization
• Read replicas
• Connection-pool tuning
• Caching
• Partitioning
• Database sharding

Now database utilization falls.

But suddenly:

Redis = 95%

Another bottleneck appears.

That's bottleneck migration.

14. The Bottleneck Chain

A useful way to think about this is:

Every layer has a maximum sustainable capacity.

For example:

Application → 20,000 RPS
Redis       → 50,000 RPS
Database    → 8,000 RPS
Payment API → 5,000 RPS

The overall system can't magically process:

20,000 RPS

just because the application servers can.

The effective capacity is constrained by the dependency chain.

In this example, the payment service might be the ultimate constraint for payment operations.

15. Why Stateless Architecture Is So Powerful

Let's return to stateless application servers.

Suppose you have:

Any request can go to any server.

Now autoscaling becomes straightforward.

Traffic increases:

5,000 RPS

You add servers:

3 → 6

Traffic decreases:

1,000 RPS

You remove servers:

6 → 2

Because application instances don't contain irreplaceable state, instances can be created and destroyed freely.

This is one of the foundations of modern cloud-native architecture.

16. What Happens If Your Application Is Stateful?

Consider a stateful server:

Now the next request goes to Server B:

One workaround is sticky sessions.

The load balancer tries to keep the user connected to the same server.

For example:

This can work.

But it creates complications.

What happens when:

Server A crashes?

The user's state may disappear.

What happens when:

Server A becomes overloaded?

The load balancer may have difficulty distributing the user's workload elsewhere.

What happens when:

You need to deploy Server A?

You have to carefully migrate active sessions.

Stateful application servers therefore make horizontal scaling more complicated.

17. Moving State Out of the Application

A common architecture is:

Now application instances can remain disposable.

The server doesn't need to remember:

"This user must come back to me."

Instead:

"Give me the state I need from shared infrastructure."

This architectural pattern is extremely common in scalable backend systems.

18. But Shared State Has a Cost

Moving state to shared infrastructure solves one problem.

It introduces another.

Your architecture now depends heavily on:

• Redis
• Database
• Object Storage
• Message Broker

Those systems must themselves be:

• Highly available
• Scalable
• Monitored
• Capacity planned
• Fault tolerant

This leads to an important principle:

Removing state from application servers doesn't remove state from the system. It relocates it.

And wherever you relocate it, you must account for its capacity and failure modes.

19. The Difference Between Scaling and Removing Work

There's another important technique that often gets overlooked.

Instead of asking:

"How do I process more work?"

ask:

"Can I avoid doing this work at all?"

For example:

Without caching:

With caching:

You didn't necessarily make the database faster.

You reduced the work it has to perform.

This is often more effective than simply adding infrastructure.

20. Finding the Actual Limiting Component

When a system doesn't scale, don't immediately add servers.

Measure.

Look at:

Application layer

• CPU utilization
• Memory utilization
• Thread pools
• Worker pools
• Request queues
• Connection pools
• Response latency

Database

• CPU
• Memory
• Connections
• Query latency
• Locks
• IOPS
• Replication lag
• Slow queries

Cache

• CPU
• Memory
• Hit rate
• Evictions
• Network throughput
• Command latency

Network

• Bandwidth
• Packet rate
• Connection count
• Network latency

External services

• Rate limits
• API latency
• Error rate
• Quotas
• Connection limits

Then ask:

Which resource approaches saturation first as traffic increases?

That is usually the most important question.

21. Don't Confuse High Utilization With the Bottleneck

Suppose you see:

Application CPU = 90%
Database CPU    = 40%

It might seem obvious that the application is the bottleneck.

But what if the application is waiting on database connections?

Maybe:

DB connections = 100%
CPU            = 40%

The database isn't CPU-bound.

The connection pool is the constraint.

Similarly:

CPU     = 40%
Memory  = 40%
Network = 99%

The application isn't CPU constrained.

It's network constrained.

Therefore:

Always identify the resource that is actually limiting throughput—not simply the resource with the highest percentage.

22. A Better Scaling Process

Instead of:

use this process:

This is a much more reliable approach to scaling distributed systems.

23. Horizontal Scaling Doesn't Mean Infinite Scaling

Horizontal scaling is powerful.

But it isn't magic.

Imagine:

1 App Server  → 1,000 RPS
2 App Servers → 2,000 RPS
4 App Servers → 4,000 RPS
8 App Servers → 7,000 RPS
16 App Servers → 8,000 RPS

Why does growth flatten?

Because something else becomes limiting.

Maybe:

• Database
• Network
• Cache
• External API
• Connection pool

This is why scalable system design requires understanding the entire request path.

24. The Most Important System Design Question

When someone asks:

"How would you scale this application?"

Don't immediately answer:

"Add a load balancer and horizontally scale the servers."

That's only part of the answer.

Instead, walk through:

Step 1 — Understand the workload

How much traffic?

• RPS
• Concurrent requests
• Data volume
• Peak traffic

Step 2 — Understand the request path

Step 3 — Identify dependencies

What does each request touch?

Step 4 — Find the limiting component

What saturates first?

Step 5 — Scale the constraint

Only then decide whether you need:

• More application servers
• Better load balancing
• Caching
• Database replicas
• Database partitioning
• Queue-based processing
• Rate limiting
• Connection pooling
• Service decomposition

That's actual scalability engineering.

25. The Mental Model

Remember this:

When you add application servers, you're increasing capacity at one layer.

The rest of the architecture still has to handle the additional work.

So the real question isn't:

"How many servers can we add?"

It's:

"Which component limits the system's ability to process more work?"

26. Final Takeaway

Horizontal scaling is one of the most important techniques in scalable system design.

Load balancing allows incoming requests to be distributed across multiple application servers.

Stateless application servers make this distribution much easier because any instance can handle any request.

But the application layer is only one part of the system.

Your servers still depend on:

• Databases
• Caches
• Message queues
• Storage
• Networks
• External APIs

These shared dependencies can become bottlenecks.

And when you scale one layer, the bottleneck may simply migrate somewhere else.

That's why:

Adding more servers won't save you if the actual bottleneck is somewhere else.

Good system design isn't about adding infrastructure everywhere.

It's about identifying the limiting component, understanding why it is limiting the system, and increasing capacity at the right layer.

The best scalability engineers don't ask:

"How can I add more servers?"

They ask:

"What's stopping me from processing more work?"

That's the mindset you need to design systems that actually scale.

Practice This Before Your Next System Design Interview

Knowing the concepts is one thing.

Being able to explain them clearly under interview pressure is another.

That's exactly why I built Intervues — a voice interview engine focused on practicing technical interviews by actually speaking through your answers. See pricing for credit packs.

No AI copilot feeding you answers.

Just you, the question, and the ability to explain your reasoning.

Practice the interview before the interview.

Visit https://intervues.club and start practicing.